KR102428920B1 - Image display device and operating method for the same - Google Patents

Abstract

a display unit, a memory storing one or more instructions, and a processor executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to obtain a plurality of images, and uses a feature extraction model to obtain a plurality of images. Extract deep features for images, classify a plurality of images into predetermined groups using the extracted deep features and classification model, display the classified results on a display unit, Disclosed is an electronic device that determines whether an update of a feature extraction model and a classification model is necessary using a result, and learns and updates at least one of a feature extraction model and a classification model according to the determination result.
The electronic device may estimate the deep feature of the image using a rule-based or artificial intelligence algorithm. When estimating a deep feature of an image using an artificial intelligence algorithm, the electronic device may use a machine learning, a neural network, or a deep learning algorithm.

Description

Electronic device and its operating method {Image display device and operating method for the same}

Various embodiments relate to an electronic device and an operating method thereof, and more particularly, to an electronic device capable of classifying a plurality of images into a predetermined group or assigning a specific keyword to a predetermined group, and an operating method thereof.

In addition, the present disclosure relates to an artificial intelligence (AI) system for simulating functions such as cognition and judgment of the human brain by using a machine learning algorithm such as deep learning and an application technology thereof.

With the development of information and communication technology and semiconductor technology, various electronic devices are developing into multimedia devices that provide various multimedia services. For example, electronic devices provide various multimedia services such as a messenger service, a broadcasting service, a wireless Internet service, a camera service, and a music reproduction service.

In addition, the electronic device provides a function for classifying and searching images. The electronic device may classify the user's images into a predetermined group by using the preset classification criterion, but there is a problem in that it cannot provide the user with an optimized classification result by using the consistent classification criterion.

Also, the electronic device may store user images together with keywords, and provides image search using keywords. However, when searching for an image using a keyword, there is a problem in that the keyword corresponding to the image to be searched must be accurately remembered because only images stored in correspondence with the keyword designated by the user are searched.

In addition, artificial intelligence systems have recently been introduced into the image processing field.

An artificial intelligence system is a computer system that implements human-level intelligence, and unlike the existing rule-based smart system, the machine learns, judges, and becomes smarter by itself. The more the AI system is used, the better the recognition rate and the more accurate understanding of user preferences, and the existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

Artificial intelligence technology consists of machine learning (deep learning) and element technologies using machine learning.

Machine learning is an algorithm technology that categorizes/learns the characteristics of input data by itself, and element technology uses machine learning algorithms such as deep learning to simulate functions such as cognition and judgment of the human brain. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying/processing human language/text, and includes natural language processing, machine translation, dialogue system, question and answer, and speech recognition/synthesis. Visual understanding is a technology that recognizes and processes objects as if they were human eyes, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, and the like. Inferential prediction is a technology for logically reasoning and predicting by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation/classification) and knowledge management (data utilization). Motion control is a technology for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), manipulation control (action control), and the like.

Various embodiments may provide an electronic device capable of extracting a characteristic of an image, classifying an image based on the characteristic of the extracted image, and searching for similar images based on the characteristic of the extracted image, and an operating method thereof have.

In addition, various embodiments may provide an electronic device capable of assigning a keyword suitable for the characteristics of an image without a user setting, and an operating method thereof.

An electronic device according to an embodiment includes a display unit, a memory for storing one or more instructions, and a processor for executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions. Obtaining images, extracting deep features for the plurality of images using a feature extraction model, and dividing the plurality of images into predetermined groups using the extracted deep features and a classification model classify, display the classified result on the display unit, determine whether to update the feature extraction model and the classification model using the classified result, and determine whether the feature extraction model and the classification model need to be updated, and according to the determination result, the feature extraction model and the classification At least one of the models can be trained and updated.

A method of operating an electronic device according to an embodiment includes acquiring a plurality of images, extracting deep features for the plurality of images using a feature extraction model, and extracting deep features. and classifying the plurality of images into predetermined groups using a classification model, displaying the classified result, and determining whether to update the feature extraction model and the classification model using the classified result and learning and updating at least one of the feature extraction model and the classification model according to the determination result.

The electronic device according to an embodiment may classify and search a plurality of images using a classification criterion optimized for each user by using a feature extraction model and a classification model learned based on user data.

When searching for an image using the electronic device according to an embodiment, it is possible to search for a desired image without memorizing and inputting keywords one by one, so that the user's convenience can be improved.

The electronic device according to an embodiment may automatically assign a keyword suitable for a characteristic of an image through interworking with a language model without a user designating a keyword.

1 is a diagram illustrating a method of classifying images in an electronic device, according to an embodiment.
2 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.
3 is a diagram referenced to describe a method for extracting deep features of an image according to an exemplary embodiment.
4 is a diagram illustrating a result of an electronic device classifying a plurality of images by using a feature extraction model and a classification model learned using general-purpose data, according to an embodiment.
5 is a diagram illustrating a result of classifying a plurality of images by using an updated feature extraction model and a classification model by an electronic device according to an exemplary embodiment;
6 is a diagram referenced to describe a method of classifying a plurality of images by a feature extraction model and a classification model learned using user data according to an embodiment.
7A is a flowchart illustrating operations of a server and an electronic device according to an exemplary embodiment.
7B is a flowchart illustrating a method of operating a server and a first processor and a second processor according to an exemplary embodiment.
7C is a flowchart illustrating an operation method of a server, a first processor, a second processor, and a third processor according to an exemplary embodiment.
8A is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.
8B is a flowchart illustrating a method of operating a first processor and a second processor included in an electronic device according to an exemplary embodiment.
8C is a flowchart illustrating an operation method of a first processor, a second processor, and a third processor included in an electronic device according to an exemplary embodiment.
9 and 10 are diagrams referenced to describe a method for an electronic device to search for an image according to an exemplary embodiment.
11 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.
12 is a block diagram of a processor 120 in accordance with some embodiments.
13 is a block diagram of a data learning unit 1300 according to some exemplary embodiments.
14 is a block diagram of a data classification unit 1400 according to some exemplary embodiments.
15 is a diagram illustrating an example of learning and recognizing data by interworking with an electronic device and a server according to an embodiment.
16 is a block diagram illustrating a configuration of an electronic device according to another exemplary embodiment.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

In the entire specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram illustrating a method of classifying images in an electronic device, according to an embodiment.

The electronic device 100 according to an embodiment may be implemented in various forms. For example, the electronic device 100 may include a mobile phone, a smart phone, a laptop computer, a desktop, a tablet PC, an e-book terminal, a digital broadcasting terminal, a personal digital assistant (PDA), or a portable computer (PMP). Multimedia Player), navigation, MP3 player, digital camera, camcorder, IPTV (Internet Protocol Television), DTV (Digital Television), wearable device (eg, smart watch), smart glass (smart glass) ), etc.) may be implemented in various electronic devices. However, the present invention is not limited thereto.

In the embodiment of the present specification, the term “user” refers to a person who controls a function or operation of an electronic device, and may include a viewer, an administrator, or an installer.

The electronic device 100 according to an embodiment may acquire a plurality of images 10 . The plurality of images 10 may include an image photographed using the electronic device 100 , an image stored in the electronic device, or an image received from an external device.

The electronic device according to an embodiment may extract deep features of a plurality of images by using the feature extraction model 20 .

The feature extraction model 20 may be a model based on a neural network. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN) may be used as the feature extraction model 20 , but is not limited thereto.

Also, the initial feature extraction model 20 may be a model learned using general-purpose data. The deep feature of the image may include a vector extracted from at least one layer included in at least one neural network by inputting the image into at least one neural network. .

A vector may include, for example, features of an image. The characteristics of the image may include, for example, the shape and type of an object included in the image, and a location where the image was captured.

Accordingly, the feature extraction model 20 may be learned using an image and image characteristics (eg, an object shape, an object type, a scene recognition result, a shooting location, etc.) as learning data. Specifically, it can be learned using an image of a dog, a type of dog, and a shooting location as learning data. In addition, it can be learned by using an image of a night view, a name of a building included in the night view, and a photographing location as learning data. Accordingly, when an image is input, the electronic device 100 may extract deep features including the features of the above-described image using the feature extraction model 20 .

When deep features for a plurality of images are extracted, the electronic device 100 according to an embodiment may classify the plurality of images by using the extracted deep features and the classification model 30 .

The classification model 30 may be a model based on a neural network. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN) may be used as the classification model 30 , but is not limited thereto.

Also, the initial classification model 30 may be a model learned using general-purpose data. For example, the classification model 30 may be trained using an image, a deep feature extracted from the image, and an image classification result as training data. Specifically, it can be learned using an image of a dog, deep features of the image (eg, dog shape, type, etc.) and a classification list of images (eg, dog, beagle, poodle, etc.) as learning data. have. In addition, an image of a night scene, a deep feature extracted from the image (eg, the location where the night scene was taken, the name of a building, etc.) and the classification of the image (eg, scenery, night view, etc.) can

Accordingly, the classification model 30 may classify the plurality of images into a predetermined group based on the similarity of the deep features to the plurality of images. In this case, the similarity of the deep features may be expressed as a distance between vectors extracted as the deep features. For example, when vectors are displayed in coordinates, the similarity may be greater as the distance between the vectors indicated in the coordinates is shorter, and the similarity may be lower as the distance between the vectors is greater. However, the present invention is not limited thereto.

The classification model 30 according to an embodiment may classify images corresponding to vectors in a preset distance range into the same group. For example, among the plurality of images, images displayed in the first area of the coordinate area (eg, images representing characteristics of 'food') are grouped into the first group 41 and displayed in the second area. Images (eg, images indicating the characteristics of 'baby') are grouped into the second group 42, and images displayed in the third area (eg, images indicating characteristics of 'top') are grouped together. It may be classified into a third group 43 .

Also, according to various embodiments, the classification model 30 may represent a short distance between vectors including shapes of dogs such as beagle and poodle. In addition, the classification model 30 may represent a short distance between vectors including a landscape, a night view, a building, and the like. On the other hand, the classification model 30 may represent a long distance between the vectors including the above-described shapes of the dog and vectors including the landscape, night view, and the like. However, the present invention is not limited thereto.

The feature extraction model 20 and the classification model 30 according to an embodiment may be configured with the same neural network or may be independently configured with different neural networks.

The feature extraction model 20 and the classification model 30 according to an embodiment may determine whether an existing feature extraction model and a classification model need to be updated by using the classified result. When it is determined that an update is necessary, it may be re-learned using user data.

For example, through supervised learning in which images classified into a predetermined group are input to at least one of the feature extraction model 20 and the classification model 30, the feature extraction model 20 and the classification model At least one of (30) can be learned. When the feature extraction model 20 and the classification model 30 are trained through supervised learning, a plurality of images generated by the user and the user's keyword mapping information corresponding to each image (eg, an image input by the user) ), the feature extraction model 20 and the classification model 30 may be trained.

Alternatively, at least of the feature extraction model 20 and the classification model 30, through unsupervised learning of re-learning the image classifier without any guidance and linking the learning results of the language model to discover the image classification criteria. One can learn In addition, at least one of the feature extraction model 20 and the classification model 30 may be trained by linking the results learned through supervised learning with the learning results of the language model and discovering the image classification criteria without guidance.

Also, the feature extraction model 20 and the classification model 30 may be trained through reinforcement learning using feedback on whether the classification result of the image according to the learning is correct. However, the present invention is not limited thereto.

2 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.

Referring to FIG. 2 , the electronic device 100 according to an embodiment may acquire a plurality of images ( S210 ).

The electronic device 100 according to an embodiment may extract deep features for a plurality of images by using the feature extraction model (S220).

The feature extraction model may be a model based on a neural network. For example, a model such as a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), or a Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as the feature extraction model, but is not limited thereto. The deep feature of the image may include a vector value extracted from at least one layer included in at least one neural network by inputting the image to at least one neural network.

Further, the deep features of an image may be stored in a metadata format such as an Exchange Image File (EXIF) for the image. Also, when the format of the image file is not JPEG, the image file can be converted to a JPEG file and the deep features of the image can be stored as EXIF information. However, the present invention is not limited thereto.

Also, since the electronic device 100 stores the deep features of the image as metadata of the image, the image can be classified using the stored deep features without re-extracting the deep features every time the image is classified. . Accordingly, the image classification speed may be increased.

In addition, when the deep feature of the image is stored in the EXIF format, even if the image file is stored in another electronic device, the deep feature information of the image may be maintained. For example, even when the image file is transmitted to an external device other than the electronic device 100 , deep feature information of the image may be maintained in the external device. Accordingly, another device can classify or search for an image by using the deep feature information of the image, and when the feature extraction model and the classification model exist in the first electronic device and the second electronic device, respectively, the first electronic device Images may be classified in the second electronic device by using the deep features of the images extracted from .

The electronic device 100 according to an embodiment may classify the plurality of images based on the deep features and the classification model of the plurality of images ( S230 ).

The classification model may be a model based on a neural network. For example, a model such as a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as the classification model, but is not limited thereto. Also, the initial classification model may be a model learned using general-purpose data. The classification model may classify the plurality of images into a predetermined group based on the similarity of the deep features to the plurality of images. In this case, the similarity of the deep features may be expressed as a distance between vectors extracted as the deep features.

Also, the electronic device 100 according to an embodiment may display a result of classification of a plurality of images.

The electronic device 100 according to an embodiment may determine whether the feature extraction model and the classification model need to be updated by using a result of the classification of a plurality of images ( S235 ).

For example, the electronic device 100 may determine whether to update the feature extraction model and the classification model based on the balance of the number of images included in each of the groups into which the plurality of images are classified. As a result of the classification, when images are included in only a specific group and the remaining groups do not include images or contain less than a preset number of images, the electronic device 100 determines that the feature extraction model and the classification model need to be updated. can On the other hand, as a result of the classification, when a predetermined number or more of images are included in the predetermined groups, the electronic device 100 may determine that the update of the feature extraction model and the classification model is not required. However, the present invention is not limited thereto, and whether the feature extraction model and the classification model need to be updated may be determined according to various criteria.

When it is determined that the update of the feature extraction model and the classification model is required, the electronic device 100 according to an embodiment may learn and update at least one of the feature extraction model and the classification model ( S240 ).

The electronic device 100 according to an embodiment may re-learn and update at least one of a feature extraction model and a classification model using user data (eg, a plurality of user images). When the feature extraction model and the classification model are updated, since user data is utilized, the feature extraction model and the classification model may be updated to suit the user data.

The electronic device 100 according to an embodiment may update at least one of a feature extraction model and a classification model periodically or upon a user's request. Alternatively, when the electronic device 100 is in a preset state, at least one of a feature extraction model and a classification model may be updated. For example, when the electronic device 100 enters a standby mode or is in a charging state, or when the electronic device 100 is connected to a Wi-Fi network, one of a feature extraction model and a classification model At least one can be updated. However, the present invention is not limited thereto.

The electronic device 100 according to an embodiment may extract deep features of a plurality of acquired images and classify the plurality of images by using the feature extraction model and the classification model updated through learning. Also, the electronic device 100 may re-extract deep features of a plurality of pre-classified images. Accordingly, deep feature information of a plurality of pre-classified images may be updated or added. Also, the electronic device 100 may reclassify the plurality of images based on the re-extracted deep features.

3 is a diagram referenced to describe a method for extracting deep features of an image according to an exemplary embodiment.

The deep feature of the image according to an embodiment may include a vector extracted from at least one layer included in at least one neural network by inputting the image to at least one neural network.

Referring to FIG. 3A , according to an embodiment, the electronic device 100 may extract a plurality of deep features of an image by inputting images into different types of neural networks. For example, by inputting an image to the first neural network 310 , extracting a first deep feature 1 from an n-th layer of the first neural network 310 , and extracting the image to the second Input to the neural network 320, extract the second deep feature 2 from the n-th layer of the second neural network 320, and input the image to the third neural network 330, A third deep feature 3 may be extracted from an n-th layer of the third neural network.

Also, referring to FIG. 3B , the electronic device 100 according to an embodiment inputs an image to one neural network, and provides information about the image in different sub-networks included in one neural network. A plurality of deep features may be extracted. For example, an image is input to a neural network including a first sub-network 340 and a second sub-network 350, and a first deep feature 1 in the n-th layer of the first sub-network 340 , and a second deep feature 2 may be extracted from the n-th layer of the second sub-network 350 .

Also, referring to FIG. 3C , the electronic device 100 according to an embodiment may input an image into one neural network and extract a plurality of deep features of the image from different layers. For example, by inputting an image into one neural network, the first deep feature 1 is extracted from the n-th layer of one neural network, and the second deep feature 2 is extracted from the m-th layer. can

The electronic device 100 may store the extracted deep feature together with neural network information from which the deep feature is extracted, neural network layer information, and sub-network information. The electronic device 100 according to an embodiment may classify a plurality of images or search for images by using deep features extracted from the same neural network.

4 is a diagram illustrating a result of an electronic device classifying a plurality of images using a feature extraction model and a classification model learned using general-purpose data, according to an embodiment.

Referring to FIG. 4 , the electronic device 100 according to an embodiment may acquire a plurality of images. For example, as shown in FIG. 4 , the electronic device 100 may acquire first to tenth images A1, A2, A3, A4, A5, A6, A7, A8, A9, and A10. have. The electronic device 100 may classify the plurality of acquired images into a predetermined group by using the feature extraction model and the classification model. In this case, the feature extraction model and classification model used may be pre-trained models using general-purpose data. For example, the feature extraction model and classification model are models trained to classify a plurality of images into six categories: 'person', 'dog', 'landscape', 'document', 'food', and 'road'. However, the present invention is not limited thereto. For example, the type and number of categories to be classified may be determined by learning the feature extraction model and the classification model, or may be set based on a user input.

When the electronic device 100 classifies a plurality of images using the feature extraction model and the classification model, first to fifth images A1, A2, A3, A4, A5, seventh to eighth images A7, A8), the tenth image A10 may be classified into a category of 'dog', and the sixth image A6 and ninth image A10 may be classified into a category of 'person'. When classifying a plurality of images according to the feature extraction model and classification model learned using general data, only the 'dog' and 'person' categories are actually used, and 'landscape', 'food', 'road', 'document' Categories such as ' will not be used. Accordingly, for users who do not use images related to 'landscape', 'food', 'road', and 'document', but mainly use images related to 'dogs' and 'people', The perceptual performance of the feature extraction model and the classification model is reduced.

5 is a diagram illustrating a result of classifying a plurality of images by using an updated feature extraction model and a classification model by an electronic device according to an exemplary embodiment;

The electronic device 100 according to an embodiment may learn and update at least one of a feature extraction model and a classification model using user images. For example, the electronic device 100 may determine whether the feature extraction model and the classification model need to be updated by using the classification results of the first to tenth images A1 to A10 of FIG. 4 . Also, the electronic device 100 may learn and update at least one of a feature extraction model and a classification model using user images.

Referring to FIG. 5 , the electronic device 100 may reclassify pre-classified images by using the updated feature extraction model and classification model. For example, as a result of classifying a plurality of images (first to tenth images A1 to A10) using the updated feature extraction model and classification model, the first image A1 and the second image A2 ), the eighth image (A8) is the first group, the third image (A3), the fourth image (A4), the sixth image (A6), and the ninth image (A9) are the second group, and the fifth image (A5) and the ninth image (A9) are the third group, the fifth image (A5), the seventh image (A7), and the tenth image (A10) are the fourth group, and the sixth to eighth images (A6) to A8) and tenth images A10 may be classified into a fifth group, and first to fifth images A1 to A5 and ninth images A9 may be classified into a sixth group. In this case, the images may be classified as being overlapped in a plurality of groups.

Compared with the classification result of FIG. 4 , the classification result may be classified into six groups in FIG. 5 , compared to two groups in FIG. 4 . Compared to the feature extraction model and classification model learned based on general-purpose data, the feature extraction model and classification model learned based on user data may classify user images more diversely.

Also, the electronic device 100 may automatically generate group names of groups into which a plurality of images are classified, or based on a user input. The electronic device 100 may automatically generate group names of groups by using the language model. The electronic device 100 may detect a keyword corresponding to the image by comparing the similarity between keywords in the language model and the similarity between deep features of the images.

For example, when a keyword (or tag) of “Beagle” is designated for the first group, the group name of the first group may be set to “Beagle”. At this time, using the language model, "Beagle" and a second keyword having a similarity (distance between deep features) between the deep feature of the image included in the first group and the deep feature of the image included in the second group can be decided The determined second keyword may be designated as a group name for the second group. Alternatively, the electronic device 100 may correspond to the first group, the second group, and the third group based on information that the first group, the second group, and the third group are groups corresponding to “dogs” in FIG. 4 . group name can be set. For example, the electronic device 100 determines the similarity and language model between the deep features of the image included in the first group, the deep features of the image included in the second group, and the deep features of the image included in the third group. By comparing the similarity between the sub-keywords of , "dog", the deep features of the image included in the first group, the deep features of the image included in the second group, and the deep features of the image included in the third group, respectively. A corresponding keyword may be determined. When keywords are determined, the electronic device 100 may designate the determined keywords as a group name of a corresponding group. Accordingly, the first group is "Beagle", the second group is "Poodle", the third group is "Anny", the fourth group is "Golden Retriever", the fifth group is "Indoor", and the sixth group is "Poodle". The group may be created with a group name of “Outdoor”. Also, the electronic device 100 may create a folder corresponding to each of the groups and store images classified into the same group in the same folder. Also, the electronic device 100 may determine keywords corresponding to each of the images included in the group.

6 is a diagram referenced to describe a method of classifying a plurality of images by a feature extraction model and a classification model learned using user data according to an embodiment.

6A is a diagram illustrating a result of classifying a plurality of images by a feature extraction model and a classification model learned using data of a first user.

Referring to FIG. 6A , the images of the first user may include first to eleventh images B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, and B11. have. In this case, the images of the first user are images obtained from the first user device, and may be images stored in the first user device, images captured by the first user device, or images received from an external device.

The first user device according to an embodiment uses a first feature extraction model and a first classification model to obtain the first user images B1, B2, B3, B4, B5, B6, B7, B8, B9, B10. , B11) can be classified into a predetermined group. The first feature extraction model and the first classification model may be a feature extraction model and a classification model learned using the first user images.

The first feature extraction model may extract a deep feature of each of the first user images, and the first classification model may classify the first user images into a predetermined group based on the deep features of each of the first user images. . In this case, the type or number of predetermined groups into which the first user images are classified may be determined by learning the first feature extraction model and the first classification model based on user data, or may be set based on a user input. For example, among the first user images, the first and second images B1 and B2 are the first group, the seventh and eighth images B7 and B8 are the second group, and the third and third The sixth and ninth images B3, B6, and B9 are a third group, the tenth and eleventh images B10 and B11 are a fourth group, and the fourth and fifth images B4 and B5 are may be classified into a fifth group, and the third and ninth images B3 and B9 may be classified into a sixth group.

Also, the first user device may automatically generate group names of groups into which a plurality of images are classified, or based on a user input. For example, the first user device may automatically generate a group name or keyword by using deep feature information and language models of images included in the classified groups.

For example, group 1 is "food", group 2 is "dogs", group 3 is "landscape", group 4 is "people", group 5 is "documents", group 6 is A group may be created with a group name of "road".

FIG. 6B is a diagram illustrating a result of classifying a plurality of images by a feature extraction model and a classification model learned using data of a second user.

Referring to FIG. 6B , the images of the second user may include first to eleventh images C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, and C11. have. In this case, the images of the second user are images obtained from the second user device, and may be images stored in the second user device, images captured by the second user device, or images received from an external device.

The second user device according to an embodiment uses the second feature extraction model and the second classification model to obtain the second user images C1, C2, C3, C4, C5, C6, C7, C8, C9, C10. , C11) can be classified into a predetermined group. The second feature extraction model and the second classification model may be a feature extraction model and a classification model learned using the second user images.

The second feature extraction model may extract deep features of each of the second user images, and the second classification model may classify the second user images into predetermined groups based on the deep features of each of the second user images. . In this case, the type or number of predetermined groups into which the second user images are classified may be determined by learning the second feature extraction model and the second classification model based on user data, or may be set based on a user input. For example, among the second user images, the first, third, and seventh images C1, C3, and C7 are in a first group, and the second and ninth images C2 and C9 are in the second group. , the third and sixth images C3 and C6 are the third group, the fourth and fifth images C4 and C5 are the fourth group, and the seventh and eighth images C7 and C8 are may be classified into a fifth group, the tenth and eleventh images C10 and C11 may be classified into a sixth group, and the first, fifth, and eleventh images C1, C5, and C11 may be classified into a seventh group. .

Also, the second user device may automatically generate group names of groups into which a plurality of images are classified, or based on a user input. The second user device may automatically generate a group name or keyword by using the deep feature information and language model of images included in the classified groups.

For example, the first group is “road”, the second group is “river”, the third group is “hanok”, the fourth group is “beach”, the fifth group is “forest”, A group name may be created for the 6th group as “building” and the 7th group as “sky”.

As described with reference to FIGS. 6A and 6B , the feature extraction model and the classification model according to an embodiment may be learned and updated using user data.

The updated feature extraction model and classification model according to an embodiment may reclassify a plurality of images included in a first group among pre-classified groups into two or more groups, or a first group and a second group among pre-classified groups. It is possible to reclassify a plurality of images included in the same group. For example, the first feature extraction model and the first classification model optimized for the first user images may classify the images B3 and B6 into the same group (“landscape” group). On the other hand, the second feature extraction model and the second classification model, which are optimized for the second user images, assign the same image C1 as image B3 and image C5 identical to image B6 to different groups (“road” group and “beach” group). can be classified as

Accordingly, the electronic device 100 according to an embodiment may provide a feature extraction model and a classification model optimized for each user, so that a plurality of images are classified based on a classification criterion optimized for each user rather than classifying a plurality of images by the same classification criterion. images can be classified.

7A is a flowchart illustrating a method of operating a server and an electronic device according to an exemplary embodiment.

Referring to FIG. 7 , the server 200 may learn a feature extraction model and a classification model using general-purpose data ( S710 ).

For example, the server 200 may learn a criterion for how to classify images based on general data. The server 200 may learn a criterion for extracting deep features of a plurality of images based on general data. Also, the server 200 may learn to classify the plurality of images into a predetermined group based on the similarity of the deep features to the plurality of images.

The electronic device 100 may receive a feature extraction model and a classification model from the server 200 (S720).

The electronic device 100 may acquire a plurality of images (S730).

The electronic device 100 may extract deep features for a plurality of images by using the feature extraction model received from the server (S740).

The electronic device 100 may classify the plurality of images based on the extracted deep features and the classification model received from the server 200 ( S750 ).

Steps 730 ( S730 ) to 750 ( S750 ) may respectively correspond to steps 210 ( S210 ) to 230 ( S230 ) of FIG. 2 .

The electronic device 100 may transmit a result obtained by classifying a plurality of images to the server 200 (S760).

The server 200 may use the classification result received from the electronic device 100 to determine whether the feature extraction model and the classification model need to be updated, and may update by learning at least one of the feature extraction model and the classification model ( S770).

The electronic device 100 may receive the updated feature extraction model and classification model learned from the server 200 (S780).

The electronic device 100 may classify a plurality of images by using the updated feature extraction model and classification model (S790).

Also, the electronic device 100 may re-extract deep features of a plurality of pre-classified images and reclassify the plurality of images based on the re-extracted deep features.

Meanwhile, each of the operations of steps 710 ( S710 ) to ( S790 ) described with reference to FIG. 7 may be performed by the electronic device 100 or the server 200 according to an embodiment. For example, in FIG. 7 , the operations of steps 710 ( S710 ) and 770 ( S770 ) are described as being performed by the server 200 , but the present invention is not limited thereto, and the operations of steps 710 and 770 ( S770 ) are not limited thereto. These may be performed in the electronic device 200 . In addition, although it has been described in FIG. 7 that the operations of steps 730 to 750 ( S750 ) and 790 ( S790 ) are performed in the electronic device 100 , the present invention is not limited thereto, and steps 730 to 750 are not limited thereto. The operations of steps S750 and 790 ( S790 ) may be performed by the server 200 .

7B is a flowchart illustrating a method of operating a server and a first processor and a second processor according to an exemplary embodiment.

Referring to FIG. 7B , the electronic device 100 may include a first processor 120a and a second processor 120b.

The first processor 120a controls execution of at least one application installed in the electronic device 100 and performs graphic processing on an image (eg, a live view image, a captured image, etc.) acquired in the electronic device 100 . can be done The first processor 120a may be implemented in the form of a system on chip (SoC) in which functions such as a central processing unit (CPU), a graphic processing unit (GPU), a communication chip, and a sensor are integrated. Also, the first processor 120a may be referred to as an application processor (AP) in this specification.

The second processor 120b may classify the images using the feature extraction model and the classification model.

Meanwhile, the second processor 120b may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI) that performs an image classification function using a data recognition model (eg, a feature extraction model and a classification model). have. According to various embodiments, in the case of a data recognition model using visual understanding as an element technology, a dedicated hardware chip for artificial intelligence (AI) may include a GPU.

Also, the electronic device 100 may further include a third processor, a fourth processor, and the like that perform the same function or a part of the function as the second processor 120b instead.

According to various embodiments of the present disclosure, the function performed by the first processor 120a may be stored in a memory and applications performing various functions may correspondingly perform the function, and the function performed by the second processor 120b may be electronic. An operating system (eg, an operation system (OS)) of the device 100 may perform the corresponding operation.

For example, a camera application may generate an image and transmit it to an OS containing a data recognition model. Also, the gallery application that displays the image may receive images extracted using the image transmitted to the OS and display the received images from the OS.

Referring to FIG. 7B , the server 200 may learn a feature extraction model and a classification model using general data ( S7110 ).

The server 200 may transmit the feature extraction model and the classification model to the electronic device 100 . For example, the electronic device 100 may be set to allow the second processor 120b to use the feature extraction model and the classification model (S7120).

The first processor 120a may acquire a plurality of images (S7130).

The first processor 120a may transmit the acquired images to the second processor 120b (S7140).

The second processor 120b may extract deep features for a plurality of images by using the feature extraction model received from the server 200 (S7150).

The second processor 120b may classify the plurality of images based on the extracted deep features and the classification model received from the server 200 (S7160).

The second processor 120b may check whether the update of the feature extraction model and the classification model is necessary, and may update by learning at least one of the feature extraction model and the classification model ( S7170 ).

The second processor 120b may classify the plurality of images by using the updated feature extraction model and classification model (S7180).

Also, the second processor 120b may re-extract deep features of the plurality of pre-classified images and reclassify the plurality of images based on the re-extracted deep features.

7C is a flowchart illustrating a server and an operating method of the first processor 120a, the second processor 120b, and the third processor 120c according to an exemplary embodiment.

Referring to FIG. 7C , the electronic device 100 may include a first processor 120a, a second processor 120b, and a third processor 120c.

Referring to FIG. 7C , the server 200 may learn a feature extraction model and a classification model using general-purpose data (S7210).

The server 200 may transmit the feature extraction model and the classification model to the electronic device 100 (S7220). For example, the electronic device 100 may be set such that the second processor 120b uses the feature extraction model and the third processor 120c uses the classification model.

The first processor 120a may acquire a plurality of images (S7230).

The first processor 120a may transmit the acquired images to the second processor 120b (S7240).

The second processor 120b may extract deep features for a plurality of images by using the feature extraction model received from the server 200 (S7250).

The second processor 120b may transmit the extracted deep features and images to the third processor 120c (S7260).

The third processor 120c may classify the plurality of images based on the deep features and the classification model (S7270).

The third processor 120c may transmit the classified result to the second processor 120b (S7280).

The second processor 120b may determine whether the feature extraction model needs to be updated, and train the feature extraction model using the classified result (S7290).

The third processor 120c may determine whether the classification model needs to be updated, and train the classification model using the classified result (S7300).

The second processor 120b may extract deep features of a plurality of images by using the updated feature extraction model (S7310). Also, the second processor 120b may re-extract the deep features of the plurality of images from which the deep features are previously extracted.

The second processor 120b may transmit the plurality of images and the extracted deep features to the third processor 120c (S7320).

The third processor 120c may classify the plurality of images using the updated classification model (S7330). Also, the third processor 120c may reclassify a plurality of pre-classified images.

8A is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.

Referring to FIG. 8A , the electronic device 100 may acquire a first image (S810). In this case, the first image may be an image captured by the electronic device or a pre-stored image. Alternatively, the first image may be an image received from an external device.

The electronic device 100 may extract a deep feature for the first image by using the feature extraction model ( S820 ). For example, the electronic device 100 may extract a deep feature of the first image by inputting the first image to at least one neural network included in the feature extraction model.

The electronic device 100 may extract at least one image having a deep feature similar to the deep feature of the first image from among pre-stored images (S830).

The electronic device 100 may extract an image similar to the first image based on the similarity between the deep feature of the first image and the deep features of a plurality of pre-stored images by using the classification model. For example, the electronic device 100 may extract an image having a difference value between the first image and the deep feature vector within a preset range from among a plurality of pre-stored images.

The electronic device 100 may display at least one extracted image on the display unit (S840).

8B is a flowchart illustrating an operation method of a first processor 120a and a second processor 120b included in an electronic device according to an exemplary embodiment.

Referring to FIG. 8B , the electronic device 100 may include a first processor 120a and a second processor 120b.

Referring to FIG. 8B , the first processor 120a may acquire a first image ( S8110 ).

The first processor 120a may transmit the first image to the second processor 120b (S8120).

The second processor 120b may extract a deep feature for the first image by using the feature extraction model ( S8130 ).

The second processor 120b may extract at least one image having a deep feature similar to the deep feature of the first image from among pre-stored images using the classification model and the deep feature ( S8140 ). For example, the second processor 120b may extract an image whose difference value from the deep feature vector of the first image is within a preset range from among a plurality of pre-stored images.

The second processor 120b may transmit the at least one extracted image to the first processor 120a (S8150).

The first processor 120a may display at least one extracted image on the display unit (S8160).

8C is a flowchart illustrating an operation method of the first processor 120a, the second processor 120b, and the third processor 120c included in the electronic device according to an exemplary embodiment.

Referring to FIG. 8C , the electronic device 100 may include a first processor 120a, a second processor 120b, and a third processor 120c.

Referring to FIG. 8C , the first processor 120a may acquire a first image ( S8210 ).

The first processor 120a may transmit the first image to the second processor 120b (S8220).

The second processor 120b may extract a deep feature for the first image by using the feature extraction model (S8230).

The second processor 120b may transmit the deep feature for the first image to the third processor 120c (S8240).

The third processor 120c may extract at least one image having a deep feature similar to the deep feature of the first image from among pre-stored images using the classification model and the deep feature (S8250). For example, the second processor 120b may extract an image whose difference value from the deep feature vector of the first image is within a preset range from among a plurality of pre-stored images.

The third processor 120c may transmit at least one extracted image to the first processor 120a (S8260).

The first processor 120a may display at least one extracted image on the display unit (S8270).

9 and 10 are diagrams referenced to describe a method for an electronic device to search for an image according to an exemplary embodiment.

Referring to FIG. 9 , the electronic device 100 according to an embodiment may acquire a first image 910 . In this case, the first image 910 may be an image captured by the electronic device 100 or a pre-stored image. Alternatively, the first image 910 may be an image received from an external device.

The electronic device 100 may extract a deep feature of the first image 910 by using the feature extraction model. The feature extraction model may include at least one neural network, and the deep feature may include a vector extracted from at least one layer by inputting the first image 910 to the at least one neural network.

In this case, the deep feature of the first image 910 may indicate characteristics of objects included in the first image 910 , for example, the first image may indicate characteristics of a 'dog'. , but not limited thereto.

The electronic device 100 may search for an image similar to the first image 910 based on the similarity between the deep feature of the first image and the deep features of the plurality of pre-stored images 920 using the classification model. have. The plurality of pre-stored images 920 may include deep features extracted from each of the images. For example, the deep feature may be stored as EXIF information of each of a plurality of images. In this case, the deep features of the first image 910 and the deep features of the plurality of images 920 may be deep features extracted using the same neural network having the same version. Alternatively, the deep feature of the first image 910 and the deep features of the plurality of images 920 may be the same sub-network of the same neural network or deep features extracted from the same layer of the same neural network.

The electronic device 100 may extract an image having a difference value of the first image from the deep feature within a preset range from among a plurality of pre-stored images 920 . For example, as shown in FIG. 9 , first to eighth images 930 among a plurality of pre-stored images 920 may be extracted and displayed as images similar to those of the first image 910 . .

When the user inputs a keyword (eg, dog, beagle, puppy, etc.) corresponding to 'dog' in order to search for 'dog' images, the electronic device 100 displays only images stored as the keyword. Search is possible, and if the user inputs a picture of 'dog' as a keyword (eg, happy) unrelated to 'dog', it is impossible to search for images stored with the keyword. Therefore, the user has the inconvenience of having to memorize the stored keywords of the images one by one and search for the corresponding images.

However, according to an embodiment, as described with reference to FIG. 9 , the electronic device extracts a deep feature of the 'dog' image using an image including 'dog' rather than a keyword, and extracts the extracted deep feature Images having a deep feature similar to ? can be searched for, so that the user can search for a desired image without having to memorize keywords one by one.

Also, referring to FIG. 10 , the electronic device 100 according to an embodiment may acquire a second image 1010 . The electronic device 100 may extract a deep feature of the second image 1010 by using the feature extraction model. The deep feature of the second image 1010 may represent a characteristic of a bridge, a characteristic of a night view, a characteristic of a river, and the like, but is not limited thereto.

The electronic device 100 may search for an image similar to the second image 1010 using the classification model, based on the similarity between the deep feature of the second image and the deep features of a plurality of pre-stored images. For example, the electronic device 100 sets the ninth to twelfth images 1030 of the pre-stored images 920 with a difference value from the deep feature of the second image within a preset range to the second image 1010 . It can be extracted and displayed as an image similar to

In order for a user to search for an image of 'night view including a bridge', related keywords (bridge, night view, etc.) must be input one by one. However, according to an exemplary embodiment, as described with reference to FIG. 10 , the electronic device uses the image 'night view including a bridge' rather than a keyword, and uses the image 'night view including a bridge' ( night view)' image can be extracted, and images having a deep feature similar to the extracted deep feature can be searched for, so that the user can search for a desired image without inputting keywords one by one.

Also, the electronic device 100 may perform an image search through keyword input. For example, when a keyword is input, the electronic device 100 may determine a keyword similar to the input keyword using a language model, and may search images corresponding to the similar keyword. When the tag name corresponding to the first image corresponds to a similar keyword, the electronic device 100 may extract the first image and display it as an image search result. In addition, when the group name into which the images are classified corresponds to a similar keyword, the second images included in the corresponding group may be extracted and displayed as an image search result. However, the present invention is not limited thereto.

11 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

Referring to FIG. 11 , the electronic device 100 according to an embodiment may include a processor 120 , a display unit 140 , and a memory 130 .

The display unit 140 according to an embodiment converts an image signal, a data signal, an OSD signal, a control signal, etc. processed by the processor 120 to generate a driving signal. The display unit 140 may be implemented as a PDP, LCD, OLED, flexible display, etc., and may also be implemented as a 3D display. Also, the display unit 140 may be configured as a touch screen and used as an input device in addition to an output device.

The display unit 140 according to an embodiment may display an image. In this case, the image displayed on the display unit 140 may be at least one of an image photographed by the electronic device 100 , an image stored in the electronic device 100 , and an image received from an external device. However, the present invention is not limited thereto.

The processor 120 according to an embodiment may execute one or more programs stored in the memory 130 . The processor 120 may include a single core, a dual core, a triple core, a quad core, and multiple cores thereof. Also, the processor 120 may include a plurality of processors. For example, the processor 120 may be implemented as a main processor (not shown) and a sub processor (not shown) operating in a sleep mode.

The memory 130 according to an embodiment may store various data, programs, or applications for driving and controlling the electronic device 100 .

Also, the program stored in the memory 130 may include one or more instructions. A program (one or more instructions) or an application stored in the memory 130 may be executed by the processor 120 .

The processor 120 according to an embodiment may obtain a plurality of images by executing one or more instructions stored in the memory 130, and extract deep features for the plurality of images by using a feature extraction model. have. For example, the feature extraction model may include a first neural network, and the processor 120 inputs each of a plurality of images to the first neural network to obtain a deep feature in at least one layer of the first neural network. Included vectors can be extracted. The processor 120 may classify the plurality of images into a predetermined group by using the extracted deep features and the classification model. For example, the classification model may include a second neural network that classifies the plurality of images into a predetermined group based on similarities between deep features of the plurality of images.

In addition, the processor 120 may store a plurality of classified images together with the deep features by executing one or more instructions stored in the memory 130 . In addition, the processor 120 controls to display the classified result on the display unit 140, and determines whether the feature extraction and classification model need to be updated, using the classified result, and according to the determination result, the feature extraction model and At least one of the classification models may be trained and updated. When at least one of the feature extraction model and the classification model is updated, the processor 120 may reclassify a plurality of pre-classified images using the updated feature extraction model and the classification model.

The processor 120 may extract a deep feature with respect to the first image, and may extract at least one image having a deep feature similar to the deep feature of the first image from among a plurality of classified images.

12 is a block diagram of a processor 120 in accordance with some embodiments.

Referring to FIG. 12 , the processor 120 according to some embodiments may include a data learning unit 1300 and a data classifying unit 1400 .

The data learning unit 1300 may learn a criterion for classifying images into a predetermined group. The data learning unit 1300 may learn a criterion for determining which data to use to classify an image into a predetermined group and how to classify the image by using the data. The data learning unit 1300 may learn a criterion for classifying an image by acquiring data to be used for learning and applying the acquired data to a data classification model to be described later. The data classification model according to an embodiment may include a feature extraction model and a classification model.

The data classification unit 1400 may classify the image based on the data. The data classification unit 1400 may classify a plurality of images into a predetermined group by using the learned data classification model. The data classification unit 1400 may classify images based on the predetermined data by obtaining predetermined data according to a preset criterion by learning, using the obtained data as an input value, and using a data classification model. . In addition, a result value output by the data classification model using the obtained data as an input value may be used to update the data classification model.

At least one of the data learning unit 1300 and the data classifying unit 1400 may be manufactured in the form of at least one hardware chip and mounted in an electronic device. For example, at least one of the data learning unit 1300 and the data classifying unit 1400 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (eg, CPU). Alternatively, it may be manufactured as a part of an application processor) or a graphics-only processor (eg, GPU) and mounted on the various electronic devices described above.

In this case, the data learning unit 1300 and the data classifying unit 1400 may be mounted on one electronic device or may be mounted on separate electronic devices, respectively. For example, one of the data learning unit 1300 and the data classifying unit 1400 may be included in the electronic device, and the other may be included in the server. In addition, the data learning unit 1300 and the data classifying unit 1400 may provide model information built by the data learning unit 1300 to the data classifying unit 1400 through a wired or wireless connection, and the data classifying unit ( Data input to 1400 may be provided to the data learning unit 1300 as additional learning data.

Meanwhile, at least one of the data learning unit 1300 and the data classifying unit 1400 may be implemented as a software module. When at least one of the data learning unit 1300 and the data classification unit 1400 is implemented as a software module (or a program module including an instruction), the software module is a computer-readable, non-transitory, non-transitory It may be stored in a readable recording medium (non-transitory computer readable media). Also, in this case, the at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

13 is a block diagram of a data learning unit 1300 according to some exemplary embodiments.

Referring to FIG. 13 , the data learning unit 1300 according to some embodiments includes a data acquiring unit 1310 , a preprocessing unit 1320 , a training data selection unit 1330 , a model learning unit 1340 , and a model evaluation unit ( 1350) may be included. However, the present invention is not limited thereto. According to various embodiments, the data learning unit 1300 may include some of the above-described components. For example, the data learning unit 1300 may include only the data acquiring unit 1310 and the model learning unit 1340 . Also, according to various embodiments, the data learning unit 1300 may further include other components in addition to the above-described components.

The data acquisition unit 1310 may acquire data necessary to classify a plurality of images into a predetermined group. The data acquisition unit 1310 may acquire data necessary for learning to classify images.

The data acquisition unit 1310 may acquire a plurality of image data. For example, the data acquiring unit 1310 may receive image data through a camera of the electronic device including the data learning unit 1300 . Alternatively, the image data may be received through an external device capable of communicating with the electronic device including the data learning unit 1300 .

The preprocessor 1320 may preprocess the acquired data so that the acquired data can be used for learning to classify an image. The preprocessor 1320 may process the acquired data into a preset format so that the model learning unit 1340, which will be described later, uses the acquired data.

The learning data selection unit 1330 may select data required for learning from among the preprocessed data. The selected data may be provided to the model learning unit 1340 . The learning data selection unit 1330 may select data necessary for learning from among preprocessed data according to a preset criterion for classifying images. In addition, the training data selection unit 1330 may select data according to a preset criterion by learning by the model learning unit 1340, which will be described later.

The model learning unit 1340 may learn a criterion for determining how to classify images based on the training data. Also, the model learning unit 1340 may learn a selection criterion for which training data should be used to classify an image.

For example, the model learning unit 1340 may learn a criterion for extracting deep features of a plurality of images, and divide the plurality of images into a predetermined group based on the similarity of the deep features to the plurality of images. can learn to classify. At this time, the similarity of deep features can be expressed as the distance between vectors extracted as deep features, and the shorter the distance between the vectors, the greater the similarity, and the greater the distance between the vectors, the lower the similarity. have. Also, a plurality of images having a distance between the deep features within a preset range may be classified into the same group.

Also, the model learning unit 1340 may train a data classification model for classifying a plurality of images by using the training data. In this case, the data classification model may be a pre-built model. For example, the data classification model may be a model built in advance by receiving basic training data (eg, sample images, etc.).

The data classification model may be constructed in consideration of the field of application of the data classification model, the purpose of learning, or the computer performance of the device. The data classification model may be, for example, a model based on a neural network. For example, a model such as a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as the data recognition model, but is not limited thereto.

According to various embodiments, when a plurality of pre-built data classification models exist, the model learning unit 1340 may determine a data classification model having a high correlation between the input training data and the basic training data as the data classification model to be trained. have. In this case, the basic learning data may be pre-classified for each data type, and the data classification model may be pre-built for each data type. For example, the basic training data is pre-classified by various criteria such as the region where the training data is generated, the time when the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, the type of object in the training data, etc. may have been

Also, the model learning unit 1340 may train the data classification model using, for example, a learning algorithm including error back-propagation or gradient descent.

Also, the model learning unit 1340 may train the data classification model through supervised learning using, for example, learning data as an input value. In addition, the model learning unit 1340 categorizes data through unsupervised learning, for example, discovering a criterion for situation determination by self-learning the type of data required for situation determination without any guidance. model can be trained. Also, the model learning unit 1340 may train the data classification model through, for example, reinforcement learning using feedback on whether the result of image classification according to learning is correct.

Also, when the data classification model is learned, the model learning unit 1340 may store the learned data classification model. In this case, the model learning unit 1340 may store the learned data classification model in the memory of the electronic device including the data classification unit 1400 . Alternatively, the model learning unit 1340 may store the learned data classification model in the memory of the electronic device including the data classification unit 1400 to be described later. Alternatively, the model learning unit 1340 may store the learned data classification model in a memory of a server connected to the electronic device through a wired or wireless network.

In this case, the memory in which the learned data classification model is stored may also store, for example, commands or data related to at least one other component of the electronic device. The memory may also store software and/or programs. A program may include, for example, a kernel, middleware, an application programming interface (API) and/or an application program (or "application"), and the like.

The model evaluator 1350 inputs evaluation data to the data classification model, and when a recognition result output from the evaluation data does not satisfy a predetermined criterion, the model learner 1340 may re-learn. In this case, the evaluation data may be preset data for evaluating the data classification model.

For example, the model evaluation unit 1350 does not satisfy a predetermined criterion when the number or ratio of evaluation data for which the classification result is not accurate among the classification results of the learned data classification model for the evaluation data exceeds a preset threshold. can be evaluated as not. For example, when the predetermined criterion is defined as a ratio of 2%, when the learned data recognition model outputs an erroneous recognition result for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluation unit 1350 learns It can be evaluated that the existing data classification model is not suitable.

On the other hand, when there are a plurality of learned data classification models, the model evaluation unit 1350 evaluates whether a predetermined criterion is satisfied with respect to each learned image classification model, and uses the model that satisfies the predetermined criterion as a final data classification model. can decide In this case, when there are a plurality of models satisfying the predetermined criteria, the model evaluator 1350 may determine any one or a predetermined number of models preset in the order of the highest evaluation score as the final data classification model.

Meanwhile, at least one of the data acquiring unit 1310 , the preprocessing unit 1320 , the training data selection unit 1330 , the model learning unit 1340 , and the model evaluation unit 1350 in the data learning unit 1300 is at least one It may be manufactured in the form of a hardware chip of For example, at least one of the data acquisition unit 1310 , the preprocessor 1320 , the training data selection unit 1330 , the model learning unit 1340 , and the model evaluation unit 1350 is artificial intelligence (AI). It may be manufactured in the form of a dedicated hardware chip for this purpose, or it may be manufactured as a part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) and mounted on the various electronic devices described above.

In addition, the data acquisition unit 1310 , the preprocessor 1320 , the training data selection unit 1330 , the model learning unit 1340 , and the model evaluation unit 1350 may be mounted in one electronic device, or separate Each of the electronic devices may be mounted. For example, some of the data acquiring unit 1310 , the preprocessing unit 1320 , the training data selection unit 1330 , the model learning unit 1340 , and the model evaluation unit 1350 are included in the electronic device, and the remaining parts are It can be included in the server.

In addition, at least one of the data acquisition unit 1310 , the preprocessor 1320 , the training data selection unit 1330 , the model learning unit 1340 , and the model evaluation unit 1350 may be implemented as a software module. At least one of the data acquisition unit 1310 , the preprocessor 1320 , the training data selection unit 1330 , the model learning unit 1340 , and the model evaluation unit 1350 includes a software module (or instruction) When implemented as a program module), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, the at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

14 is a block diagram of a data classification unit 1400 according to some exemplary embodiments.

Referring to FIG. 14 , the data classifying unit 1400 according to some embodiments includes a data obtaining unit 1410 , a preprocessing unit 1420 , a classification data selecting unit 1430 , a classification result providing unit 1440 , and a model updating unit. (1450). However, the present invention is not limited thereto. According to various embodiments, the data classification unit 1400 may include some of the above-described components. For example, the data classifying unit 1400 may include only the data obtaining unit 1410 and the classification result providing unit 1440 . Also, according to various embodiments, the data classification unit 1400 may further include other components in addition to the above-described components.

The data acquirer 1410 may acquire data required for image classification, and the preprocessor 1420 may preprocess the acquired data so that the acquired data for image classification may be used. The preprocessor 1420 may process the acquired data into a preset format so that the classification result providing unit 1440, which will be described later, uses the acquired data for image classification.

The classification data selection unit 1430 may select data required for image classification from among the preprocessed data. The selected data may be provided to the classification result providing unit 1440 . The classification data selection unit 1430 may select some or all of the preprocessed data according to a preset criterion for image classification. Also, the classification data selection unit 1430 may select data according to a preset criterion by learning by the model learning unit 1340 .

The classification result providing unit 1440 may classify the image by applying the selected data to the data classification model. The classification result providing unit 1440 may provide a classification result according to the purpose of classification of data. The classification result providing unit 1440 may apply the selected data to the data classification model by using the data selected by the classification data selecting unit 1430 as an input value. Also, the classification result may be determined by a data classification model.

For example, the classification result providing unit 1440 may provide a result in which a plurality of images are classified into a predetermined group. Images classified into the same group among the plurality of images may be stored in the same folder.

Also, according to various embodiments, the classification result providing unit 1440 may estimate another image similar to one image. For example, the classification result providing unit 1440 may estimate an image having a deep feature similar to the first image (eg, one of pre-stored images or one of newly input images).

The model updating unit 1450 may update the data classification model based on the evaluation of the classification result provided by the classification result providing unit 1440 . For example, the model update unit 1450 determines whether the data classification model needs to be updated according to the classification result provided by the classification result provision unit 1440 , and when the data classification model needs to be updated, model learning The unit 1340 may be used to cause the data classification model to be updated. The model learning unit 1340 may update the data classification model by re-learning the data classification model using the user image data.

Meanwhile, at least one of the data acquisition unit 1410 , the preprocessor 1420 , the classification data selection unit 1430 , the classification result providing unit 1440 , and the model update unit 1450 in the data classification unit 1400 is at least It may be manufactured in the form of a single hardware chip and mounted in an electronic device. For example, at least one of the data acquiring unit 1410 , the preprocessing unit 1420 , the classification data selection unit 1430 , the classification result providing unit 1440 , and the model updating unit 1450 is artificial intelligence (AI). ) may be manufactured in the form of a dedicated hardware chip, or may be manufactured as a part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) and mounted on the various electronic devices described above.

In addition, the data acquisition unit 1410 , the preprocessor 1420 , the classification data selection unit 1430 , the classification result provision unit 1440 , and the model update unit 1450 may be mounted in one electronic device or separately. It may be mounted on each of the electronic devices of For example, some of the data acquiring unit 1410 , the preprocessing unit 1420 , the classification data selecting unit 1430 , the classification result providing unit 1440 , and the model updating unit 1450 are included in the electronic device, and the remaining parts are included in the electronic device. may be included in the server.

Also, at least one of the data acquisition unit 1410 , the preprocessor 1420 , the classification data selection unit 1430 , the classification result provision unit 1440 , and the model update unit 1450 may be implemented as a software module. At least one of the data acquisition unit 1410 , the preprocessor 1420 , the classification data selection unit 1430 , the classification result provision unit 1440 , and the model update unit 1450 is a software module (or instruction) When implemented as a program module including), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, the at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

15 is a diagram illustrating an example in which an electronic device and a server learn and recognize data by interworking with each other according to an embodiment.

Referring to FIG. 15 , the server 2000 may analyze a user image to learn a criterion for classifying the image, and the electronic device 100 may select a plurality of images based on the learning result by the server 2000 . can be classified.

In this case, the model learning unit 2340 of the server 2000 may perform the function of the model learning unit 1340 illustrated in FIG. 13 . The model learning unit 2340 of the server 2000 may learn a criterion for extracting deep features of a plurality of images, and based on the similarity of the deep features to the plurality of images, the plurality of images into a predetermined group. can learn to classify. The model learning unit 2340 may learn a criterion for classifying a plurality of images by acquiring data to be used for learning and applying the acquired data to a data classification model to be described later.

Also, the classification result providing unit 1440 of the electronic device 100 may classify the plurality of images by applying the data selected by the classification data selection unit 1430 to the data classification model generated by the server 2000 . . For example, the classification result providing unit 1440 transmits the data selected by the classification data selection unit 1430 to the server 2000 , and the server 2000 classifies the data selected by the classification data selection unit 1430 . You can request to classify a plurality of images by applying to the model. Also, the classification result providing unit 1440 may provide a result of classifying a plurality of images into a predetermined group. Images classified into the same group among the plurality of images may be stored in the same folder.

Alternatively, the classification result providing unit 1440 of the electronic device 100 receives the classification model generated by the server 2000 from the server 2000, analyzes the image using the received classification model, and a plurality of images. can be classified. In this case, the classification result providing unit 1440 of the electronic device 100 may classify the plurality of images by applying the data selected by the classification data selection unit 1430 to the data classification model received from the server 2000 . . When receiving the classification model from the server 2000 and classifying a plurality of images in the electronic device 100, there is no need to transmit user data (a plurality of user images) to the server, so user data security and personal information protection are improved. can be strengthened

16 is a block diagram illustrating a configuration of an electronic device according to another exemplary embodiment. The electronic device 300 of FIG. 16 may be an embodiment of the electronic device 100 of FIG. 1 .

Referring to FIG. 16 , the electronic device 300 according to an embodiment includes a controller 330 , a sensor unit 320 , a communication unit 340 , an output unit 350 , a user input unit 360 , and an A/V input unit ( 370) and a storage unit 380 may be included.

The control unit 330 of FIG. 16 is the processor 120 of FIG. 11 ), the storage unit 380 of FIG. 16 is the memory 130 of FIG. 11 , and the display unit 351 of FIG. 16 is the display unit of FIG. 11 . Each may correspond to (140). The same contents as those described with reference to FIG. 11 will be omitted from FIG. 16 .

The communication unit 340 may include one or more components for performing communication between the electronic device 300 and an external device (eg, a server, etc.). For example, the communication unit 340 may include a short-range communication unit 341 , a mobile communication unit 342 , and a broadcast receiving unit 343 .

The short-range wireless communication unit 341 includes a Bluetooth communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, It may include a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, and the like, but is not limited thereto.

The mobile communication unit 342 transmits/receives wireless signals to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission/reception of a voice call signal, a video call signal, or a text/multimedia message.

The broadcast receiver 343 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an embodiment, the electronic device 300 may not include the broadcast receiver 343 .

The communication unit 340 may receive at least one image from an external device. Alternatively, the communication unit 340 may request and receive a feature extraction model and a classification model from an external server. The communication unit 340 may transmit a classification result of the plurality of images to an external server, and may receive an updated feature extraction model and a classification model based on the classification result.

The output unit 350 is for outputting an audio signal, a video signal, or a vibration signal, and may include a display unit 351 , a sound output unit 352 , a vibration motor 353 , and the like. Since the display unit 351 has already been described with reference to FIG. 11 , a description of the display unit 351 will be omitted.

The sound output unit 352 outputs audio data received from the communication unit 340 or stored in the storage unit 380 . Also, the sound output unit 352 outputs a sound signal related to a function (eg, a call signal reception sound, a message reception sound, and a notification sound) performed by the electronic device 100 . The sound output unit 352 may include a speaker, a buzzer, and the like.

The vibration motor 353 may output a vibration signal. For example, the vibration motor 353 may output a vibration signal corresponding to the output of audio data or video data (eg, a call signal reception sound, a message reception sound, etc.). Also, the vibration motor 353 may output a vibration signal when a touch is input to the touch screen.

The controller 330 controls the overall operation of the electronic device 300 . For example, the control unit 330 executes the programs stored in the storage unit 380 , so that the communication unit 340 , the output unit 350 , the user input unit 360 , the sensing unit 320 , and the A/V input unit ( 370) can be controlled.

The user input unit 360 means a means for a user to input data for controlling the electronic device 300 . For example, the user input unit 360 includes a key pad, a dome switch, and a touch pad (contact capacitive type, pressure resistance film type, infrared sensing method, surface ultrasonic conduction method, integral type). There may be a tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto.

The sensor unit 320 may include not only a sensor for sensing the user's biometric information, but also a sensor for detecting a state of the electronic device 300 or a state around the electronic device 300 . Also, the sensor unit 320 may transmit information sensed by the sensor to the control unit 320 .

The sensor unit 320 includes a geomagnetic sensor, an acceleration sensor, a temperature/humidity sensor, an infrared sensor, a gyroscope sensor, a location sensor (eg, GPS)), a barometric pressure sensor, a proximity sensor, and RGB It may include at least one of a sensor (illuminance sensor), but is not limited thereto. Since a function of each sensor can be intuitively inferred from a person skilled in the art from the name, a detailed description thereof will be omitted.

The A/V (Audio/Video) input unit 370 is for inputting an audio signal or a video signal, and may include a camera 371 , a microphone 372 , and the like. The camera 371 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a shooting mode. The image captured through the image sensor may be processed through the controller 330 or a separate image processing unit (not shown).

The image frame processed by the camera 371 may be stored in the storage unit 380 or transmitted to the outside through the communication unit 340 . Two or more cameras 371 may be provided according to the configuration of the electronic device 300 .

The microphone 372 receives an external sound signal and processes it as electrical voice data. For example, the microphone 372 may receive an acoustic signal from an external device or a speaker. The microphone 372 may use various noise removal algorithms for removing noise generated in the process of receiving an external sound signal.

The storage unit 380 may store a program for processing and control of the controller 330 , and may store input/output data (eg, application, content, time zone information of an external device, address book, etc.).

Storage unit 380 is a flash memory type (flash memory type), hard disk type (hard disk type), multimedia card micro type (multimedia card micro type), card type memory (for example, SD or XD memory, etc.), RAM (Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk. Also, the electronic device 300 may operate a web storage or a cloud server that performs a storage function of the storage unit 380 on the Internet.

Programs stored in the storage unit 380 may be classified into a plurality of modules according to their functions, for example, may be classified into a UI module 381 , a touch screen module 382 , a notification module 383 , and the like. have.

The UI module 381 may provide a specialized UI, GUI, etc. interworking with the electronic device 300 for each application. The touch screen module 382 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the controller 130 .

The touch screen module 382 may recognize and analyze the touch code. The touch screen module 382 may be configured as separate hardware including a controller.

The notification module 383 may generate a signal for notifying the occurrence of an event in the electronic device 300 . Examples of events generated in the electronic device 300 include call signal reception, message reception, key signal input, schedule notification, and the like. The notification module 383 may output a notification signal in the form of a video signal through the display 351 , may output a notification signal in the form of an audio signal through the sound output unit 352 , and the vibration motor 353 . It is also possible to output a notification signal in the form of a vibration signal through

Meanwhile, the block diagrams of the electronic devices 100 and 300 illustrated in FIGS. 11 and 16 are block diagrams for an exemplary embodiment. Each component of the block diagram may be integrated, added, or omitted according to specifications of the actually implemented electronic devices 100 and 300 . That is, two or more components may be combined into one component, or one component may be subdivided into two or more components as needed. In addition, the function performed in each block is for describing the embodiments, and the specific operation or device does not limit the scope of the present invention.

The method of operating an electronic device according to an embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In addition, the control method according to the disclosed embodiments may be provided included in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored. For example, computer program products may include products (eg, downloadable apps) in the form of S/W programs distributed electronically through manufacturers of electronic devices or electronic markets (eg, Google Play Store, App Store). have. For electronic distribution, at least a portion of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a SW program.

The computer program product, in a system consisting of a server and an electronic device, may include a storage medium of the server or a storage medium of the electronic device. Alternatively, when there is a third device (eg, a smartphone) that is communicatively connected to the server or the electronic device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself transmitted from the server to the electronic device or the third device, or transmitted from the third device to the electronic device.

In this case, one of the server, the electronic device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the electronic device, and the third device may execute the computer program product to distribute the method according to the disclosed embodiments.

For example, a server (eg, a cloud server or an artificial intelligence server) may execute a computer program product stored in the server to control an electronic device communicatively connected to the server to perform the method according to the disclosed embodiments.

As another example, the third device may execute a computer program product to control the electronic device communicatively connected to the third device to perform the method according to the disclosed embodiment. When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute the computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also included in the scope of the present invention. belongs to

100: electronic device

Claims (21)

In an electronic device,
display unit;
a memory storing one or more instructions; and
a processor executing the one or more instructions stored in the memory;
The processor by executing the one or more instructions,
Obtaining a plurality of images, extracting deep features for the plurality of images using a feature extraction model based on general data,
Classifying the plurality of images into predetermined groups using a classification model based on the extracted deep features and general data,
Display the classified result on the display unit,
Determining whether to update the feature extraction model and the classification model using the classified result,
When at least one of the feature extraction model and the classification model needs to be updated, by using user data of the electronic device, at least one of the feature extraction model and the classification model is trained and updated;
The processor determines whether to update at least one of the feature extraction model and the classification model based on a balance of the number of images classified into the predetermined groups.
According to claim 1,
The processor by executing the one or more instructions,
and storing the plurality of images and the deep features.
3. The method of claim 2,
The processor by executing the one or more instructions,
and storing the deep features in an exchange image file (EXIF) format for the plurality of images.
According to claim 1,
The feature extraction model includes a first neural network,
The processor by executing the one or more instructions,
Inputting each of the plurality of images to the first neural network, extracting a vector from at least one layer of the first neural network,
The deep feature for each of the plurality of images includes the extracted vector.
5. The method of claim 4,
The classification model includes a second neural network that classifies the plurality of images into a predetermined group based on the similarity between the deep features of the plurality of images.
6. The method of claim 5,
The degree of similarity between the deep features for the plurality of images is expressed as a difference between vectors included in the deep features, the greater the difference between the vectors, the smaller the similarity, and the smaller the difference between the vectors, the greater the similarity. The difference between vectors corresponding to images classified into the same group is within a preset range.
According to claim 1,
The processor by executing the one or more instructions,
acquiring a first image, extracting a deep feature of the first image, and having a deep feature similar to the deep feature of the first image among the classified plurality of images based on the deep feature of the first image Extracting at least one image, and displaying the extracted at least one image on the display unit.
According to claim 1,
The processor by executing the one or more instructions,
An electronic device that trains and updates at least one of the feature extraction model and the classification model periodically or when a user's request is input.
According to claim 1,
The processor by executing the one or more instructions,
When the classification model is updated, reclassifying the plurality of classified images using the extracted deep features and the updated classification model.
According to claim 1,
The processor by executing the one or more instructions,
When the feature extraction model is updated, deep features for the plurality of classified images are re-extracted using the updated feature extraction model, and the classified by using the re-extracted deep features and the classification model An electronic device for reclassifying a plurality of images.
According to claim 1,
When at least one of the feature extraction model and the classification model is updated,
The processor by executing the one or more instructions,
An electronic device that reclassifies the plurality of images included in the first group into two or more groups based on deep features of the plurality of images included in the first group among the group into which the plurality of images are classified .
According to claim 1,
When at least one of the feature extraction model and the classification model is updated,
The processor by executing the one or more instructions,
and reclassifying a plurality of images included in a first group and a second group among the group into which the plurality of images are classified into the same group.
According to claim 1,
The processor by executing the one or more instructions,
and generating a group name corresponding to the predetermined groups.
A method of operating an electronic device, comprising:
acquiring a plurality of images;
extracting deep features for the plurality of images using a feature extraction model based on general-purpose data;
classifying the plurality of images into predetermined groups using a classification model based on the extracted deep features and general data, and displaying the classification result;
determining whether to update the feature extraction model and the classification model by using the classified result; and
When at least one of the feature extraction model and the classification model needs to be updated, using user data of the electronic device to train and update at least one of the feature extraction model and the classification model;
Determining whether to update the feature extraction model and the classification model by using the classified result comprises:
and determining whether to update at least one of the feature extraction model and the classification model based on a balance of the number of images classified into the predetermined groups.
15. The method of claim 14,
The method of operation is
and storing the deep features in an exchange image file (EXIF) format for the plurality of images.
15. The method of claim 14,
The feature extraction model includes a first neural network, and the classification model is a second classification model for classifying the plurality of images into a predetermined group based on similarities between deep features of the plurality of images. Includes a neural network,
The step of extracting deep features for the plurality of images,
inputting each of the plurality of images to the first neural network, and extracting a vector from at least one layer of the first neural network,
Classifying the plurality of images into predetermined groups, and displaying the classified result,
and classifying at least one image having a difference between vectors within a preset range among the plurality of images into the same group.
15. The method of claim 14,
The method of operation is
acquiring a first image;
extracting a deep feature of the first image;
extracting at least one image having a deep feature similar to the deep feature of the first image from among the plurality of classified images based on the deep feature of the first image; and
The method of operating an electronic device further comprising the step of displaying the at least one extracted image.
15. The method of claim 14,
The method of operation is
When the classification model is updated, the method further comprising the step of reclassifying the plurality of classified images using the deep features and the updated classification model.
15. The method of claim 14,
The method of operation is
re-extracting deep features for the plurality of classified images when the feature extraction model is updated; and
Using the re-extracted deep features and the classification model, the method further comprising the step of reclassifying the plurality of classified images.
the electronic device,
acquiring a plurality of images;
extracting deep features for the plurality of images using a feature extraction model based on general-purpose data;
classifying the plurality of images into predetermined groups using a classification model based on the extracted deep features and general data, and displaying the classification result;
determining whether to update at least one of the feature extraction model and the classification model based on a balance of the number of images classified into the predetermined groups; and
When at least one of the feature extraction model and the classification model needs to be updated, using user data of the electronic device, training and updating at least one of the feature extraction model and the classification model is stored. computer-readable recording medium.
delete

Images