Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/51—Indexing; Data structures therefor; Storage structures
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
  • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/22—Matching criteria, e.g. proximity measures
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/23—Clustering techniques
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
  • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
  • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
  • G06F18/24133—Distances to prototypes
  • G06F18/24143—Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/045—Combinations of networks
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
  • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
  • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
  • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
  • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/044—Recurrent networks, e.g. Hopfield networks

Definitions

• the present disclosure relates generally to electronic apparatuses and methods of operating the same, and for example, to an electronic apparatus capable of classifying a plurality of images into a certain group or assigning a particular keyword to a certain group, and a method of operating the electronic apparatus.
• the present disclosure relates to an artificial intelligence (AI) system and application thereof, which provide recognition and decision-making, using a machine learning algorithm, such as deep learning.
• AI artificial intelligence
• an electronic apparatus provides various multimedia services, such as a messenger service, a broadcasting service, a wireless Internet service, a camera service, and a music reproducing service.
• multimedia services such as a messenger service, a broadcasting service, a wireless Internet service, a camera service, and a music reproducing service.
• the electronic apparatus provides a function of classifying and searching for an image.
• the electronic apparatus may classify images of a user into a certain group by using a pre-set classification criterion, but it is not possible to provide a classification result optimized for the user since a set classification criterion is used.
• the electronic apparatus may store the images of the user together with a keyword, and provides an image search function that uses the keyword.
• the electronic apparatus may store the images of the user together with a keyword, and provides an image search function that uses the keyword.
• image search function that uses the keyword.
• AI artificial intelligence
• the AI system is a system in which a machine self-learns, determines, and becomes intelligent, unlike an existing rule-based smart system.
• the more the AI system is used a recognition rate is increased and a user's taste is more accurately understood, and thus the existing rule-based smart system is gradually replaced by a deep-learning-based AI system.
• AI technology includes machine learning (deep learning) and element technologies using the machine learning.
• the machine learning is an algorithm technology that self-classifies and learns features of input data.
• Element technologies are technologies for simulating functions, such as recognition and determination, using a machine learning algorithm, such as deep-learning, and includes the technical fields of linguistic understanding, visual understanding, inference/prediction, knowledge representation, or operation control.
• Linguistic understanding is a technique of recognizing languages/characters of people and applying/processing the languages/characters, and includes natural language processing, machine translation, a dialog system, questions and answers, or voice recognition/synthesis.
• Visual understanding is a technique of recognizing an object, and includes object recognition, object tracing, image search, person recognition, scene understanding, space understanding, or image improvement.
• Inference/prediction is a technique of logically inferring and predicting information by determining the information, and includes knowledge/probability-based inference, optimization prediction, preference-based plans, or recommendation.
• Knowledge representation is a technique of automating experience information into knowledge data, and includes knowledge construction (data generation/classification) or knowledge management (data application).
• Operation control is a technique of controlling automatic driving of a vehicle or movement of a robot, and includes movement control (navigation, collision avoidance, driving) or manipulation control (action control).
• electronic apparatuses capable of extracting a feature of an image, classifying the image based on the extracted feature, and searching for similar images based on the extracted feature, and methods of operating the electronic apparatuses.
• FIG. 1 is a diagram illustrating an example method of classifying images, the method performed by an electronic apparatus, according to an example embodiment of the present disclosure
• FIG. 2 is a flowchart illustrating an example method of operating an electronic apparatus, according to an example embodiment of the present disclosure
• FIGS. 3A, 3B and 3C are diagrams illustrating an example method of extracting a deep feature of an image, according to an example embodiment of the present disclosure
• FIG. 4 is a diagram illustrating an example result of classifying, by an electronic apparatus, a plurality of images using a feature extraction model and a classification model, which are trained based on general-purpose data, according to an example embodiment of the present disclosure
• FIG. 5 is a diagram illustrating an example result of classifying, by an electronic apparatus, a plurality of images using a feature extraction model and a classifying mode, which are updated, according to an example embodiment of the present disclosure
• FIGS. 6A and 6B are diagrams illustrating an example method of classifying a plurality of images using a feature extraction model and a classification model, which are trained based on user data, according to an example embodiment of the present disclosure
• FIG. 7B is a flowchart illustrating an example method of operating a server, a first processor, and a second processor, according to an example embodiment of the present disclosure
• FIG. 7C is a flowchart illustrating an example method of operating a server, a first processor, a second processor, and a third processor, according to an example embodiment of the present disclosure
• FIG. 8C is a flowchart illustrating an example method of operating a first processor, a second processor, and a third processor included in an electronic apparatus, according to an example embodiment of the present disclosure
• FIG. 12 is a block diagram illustrating an example processor according to an example embodiment of the present disclosure.
• FIG. 14 is a block diagram illustrating an example data classifying unit according to an example embodiment of the present disclosure.
• an electronic apparatus includes: a display; a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction stored in the memory, to cause the electronic device to: obtain a plurality of images, extract deep features with respect to the plurality of images using a feature extraction model, classify the plurality of images into certain groups using the extracted deep features and a classification model, display a result of the classification on the display, determine whether the feature extraction model and the classification model need to be updated using the result of the classification, and train and update at least one of the feature extraction model and the classification model based on a result of the determination.
• the term "and/or" includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of,” when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list.
• the term "user” may refer, for example, to a person who controls functions or operations of an electronic apparatus, and may be a viewer, a manager, or an installation engineer.
• the electronic apparatus 100 may obtain a plurality of images 10.
• the plurality of images 10 may include an image captured using the electronic apparatus 100, an image stored in the electronic apparatus 10, or an image received from an external apparatus.
• the electronic apparatus 100 may extract deep features of the plurality of images 10 using a feature extraction model 20.
• the feature extraction model 20 may, for example, and without limitation, be a neural network-based model.
• a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN), may be used as the feature extraction model 20, or the like, but is not limited thereto.
• DNN deep neural network
• RNN recurrent neural network
• BBDNN bidirectional recurrent deep neural network
• a vector may include, for example, a feature of an image.
• a feature of an image may include, for example, a shape or type of an object included in the image, or a place where the image is captured.
• the feature extraction model 20 may be trained using an image and a feature (for example, a shape of an object, a type of an object, a scene recognition result, or a captured place) of the image as learning data.
• the feature extraction model 20 may be trained by using, as the learning data, an image of a puppy, a type of the puppy, and a captured place.
• the feature extraction model 20 may be trained by using, as the learning data, an image of a night view, a name of a building included in the night view, and a captured place. Accordingly, when an image is input, the electronic apparatus 100 may extract deep features including such features of the image by using the feature extraction model 20.
• the electronic apparatus 100 may classify the plurality of images 10 using the extracted deep features and a classification model 30.
• the classification model 30 may, for example, and without limitation, be a neural network-based model.
• a model such as DNN, RNN, or BRDNN, or the like, may be used as the classification model 30, but is not limited thereto.
• the classification model 30 may initially be a model trained based on general-purpose data.
• the classification model 30 may be trained using an image, a deep feature extracted from the image, and an image classification result as learning data.
• the classification model 30 may be trained by using, as the learning data, an image of a puppy, a deep feature (for example, a shape or type of the puppy) of the image, and a classification list (for example, a puppy, a beagle, or a poodle) of the image.
• the classification model 30 may be trained by using, as the learning data, an image of a night view, a deep feature (for example, a location where the night view was imaged or a name of a building) extracted from the image, and a classification (for example, a landscape or a night view) of the image.
• a deep feature for example, a location where the night view was imaged or a name of a building
• a classification for example, a landscape or a night view
• the classification model 30 may classify the plurality of images 10 into a certain group based on similarity between the deep features with respect to the plurality of images 10.
• the similarity between the deep features may, for example, be indicated by a distance between vectors extracted as the deep features.
• the similarity may be high when the distance between the vectors displayed on the coordinate system is short, and the similarity may be low when the distance between the vectors is long.
• an embodiment is not limited thereto.
• the classification model 30 may classify images corresponding to vectors within a pre-set distance range into one group. For example, from among the plurality of images 10, images (for example, images indicating features of 'food') displayed in a first region of a coordinate region may be classified into a first group 41, images (for example, images indicating features of a 'baby') displayed in a second region may be classified into a second group 42, and images (for example, images indicating features of a 'tower') displayed in a third region may be displayed into a third group 43.
• the classification model 30 may indicate a distance between vectors including shapes of puppies, such as a beagle and a poodle, to be short. Also, the classification model 30 may indicate a distance between vectors indicating a landscape, a night view, and a building to be short. Meanwhile, the classification model 30 may indicate the distance between the vectors including the shapes or puppies, and the distance between the vectors including the landscape and night view to be long.
• the various example embodiments are not limited thereto.
• the feature extraction model 20 and the classification model 30 may be configured in one neural network or may be independently configured in different neural networks.
• the feature extraction model 20 and the classification model 30 may determine whether an existing feature extraction model and an existing classification model need to be updated by using a result of classification. When it is determined that they need to be updated, the feature extraction model 20 and the classification model 30 may be re-trained by using user data.
• At least one of the feature extraction model 20 and the classification model 30 may be trained via supervised learning using the plurality of images 10 classified into the certain group as input values input to at least one of the feature extraction model 20 and the classification model 30.
• the feature extraction model 20 and the classification model 30 may be trained using a plurality of images generated by a user and keyword mapping information (for example, features of the images input by the user) of the user corresponding to each of the plurality of images.
• At least one of the feature extraction model 20 and the classification model 30 may be trained via unsupervised learning in which an image classifier is re-trained without any particular supervision and an image classification criterion is found in relation to a learning result of a language model. Also, at least one of the feature extraction model 20 and the classification model 30 may be trained by finding the image classification criterion without supervision, by associating the learning result of the language model and a result of learning via supervised learning.
• the feature extraction model 20 and the classification model 30 may be trained via reinforcement learning using feedback indicating whether a classification result of images according to learning is correct.
• an embodiment is not limited thereto.
• FIG. 2 is a flowchart illustrating an example method of operating the electronic apparatus 100, according to an example embodiment of the present disclosure.
• the electronic apparatus 100 may obtain a plurality of images, in operation S210.
• the electronic apparatus 100 may extract deep features with respect to the plurality of images using a feature extraction model, in operation S220.
• a feature extraction model may be a neural network-based model.
• a model such as DNN, RNN, or BRDNN, or the like, may be used as the feature extraction model, but is not limited thereto.
• a deep feature of an image may include a vector extracted from at least one layer included in at least one neural network by inputting the image into the at least one neural network.
• the deep feature of the image may be stored in a metadata form, such as EXIF.
• a format of an image file is not JPEG, the image file may be converted into JPEG and the deep feature of the image may be stored in EXIF.
• an embodiment is not limited thereto.
• the electronic apparatus 100 may store the deep feature of the image as metadata of the image such that the deep feature is not extracted again whenever the image is classified, and classify the image by using the stored deep feature. Accordingly, an image classification speed may be increased.
• information about the deep feature may be maintained even when the image file is stored in another electronic apparatus.
• the information about the deep feature of the image may be maintained in an external apparatus even when the image file is transmitted to the external apparatus instead of the electronic apparatus 100.
• another apparatus may classify or search for the image by using the information about the deep feature, and when a feature extraction model and a classification model are respectively stored in a first electronic apparatus and a second electronic apparatus, the second electronic apparatus may classify images by using deep features of images extracted by the first electronic apparatus.
• the electronic apparatus 100 may classify the plurality of images based on the deep features with respect to the plurality of images, and a classification model, in operation S230.
• the classification model may be a neural network-based model.
• a model such as DNN, RNN, or BRDNN, or the like, may be used as the classification model, but is not limited thereto.
• the classification model may initially be a model trained based on general-purpose data.
• the classification model may classify the plurality of images into a certain group based on similarity between the deep features with respect to the plurality of images.
• the similarity between the deep features may be indicated by a distance between vectors extracted as the deep features.
• the electronic apparatus 100 may display a result of classifying the plurality of images.
• the electronic apparatus 100 may determine whether the feature extraction model and/or the classification model need to be updated based on a balance of the numbers of images included in groups into which the plurality of images are classified. The electronic apparatus 100 may determine that the feature extraction model and/or the classification model need to be updated when, based on the result of classifying the plurality of images, images are included only in a particular group while no image is included in other groups or images numbering less than a pre-set number are included in the other groups. On the other hand, the electronic apparatus 100 may determine that the feature extraction model and/or the classification model do not need to be updated when, based on the result of classifying the plurality of images, images included in certain groups are equal to or more than the pre-set number.
• an example embodiment is not limited thereto, and necessity of updating the feature extraction model and the classification model may be determined based on various criteria.
• the electronic apparatus 100 may update at least one of the feature extraction model and the classification model periodically or when there is a user request. At least one of the feature extraction model and the classification model may be updated when the electronic apparatus 100 is in a pre-set state. For example, at least one of the feature extraction model and the classification model may be updated when the electronic apparatus 100 enters a standby mode or is in a charging state, or when the electronic apparatus 100 is connected to a Wi-Fi network.
• an embodiment is not limited thereto.
• the electronic apparatus 100 may extract the deep features of the obtained plurality of images, and classify the plurality of images by using the feature extraction model and the classification model, which are updated via learning. Also, the electronic apparatus 100 may re-extract the deep features of the plurality of images, which are pre-classified. Accordingly, the information about the deep features of the pre-classified plurality of images may be updated or added. Also, the electronic apparatus 100 may re-classify the plurality of images based on the re-extracted deep features.
• FIGS. 3A, 3B and 3C are reference diagrams illustrating an example method of extracting a deep feature of an image, according to an example embodiment of the present disclosure.
• a deep feature of an image may, for example, and without limitation, include a vector extracted from at least one layer included in at least one neural network by inputting the image into the at least one neural network.
• the electronic apparatus 100 may extract a plurality of deep features with respect to an image by inputting the image input different types of neural network.
• the image may be input to a first neural network 301 to extract a first deep feature from an n-th layer of the first neural network 301
• the image may be input to a second neural network 302 to extract a second deep feature from an n-th layer of the second neural network 302
• the image may be input to a third neural network 303 to extract a third deep feature from an n-th layer of the third neural network 303.
• the electronic apparatus 100 may extract a plurality of deep features with respect to an image from different sub-networks included in one neural network by inputting the image into the one neural network.
• the image may be input to a neural network including a first sub-network 304 and a second sub-network 305 to extract a first deep feature from an n-th layer of the first sub-network 304 and extract a second deep feature from an n-th layer of the second sub-network 305.
• the electronic apparatus 100 may extract a plurality of deep features with respect to an image from different layers by inputting the image into one neural network.
• the image may be input to the one neural network to extract a first deep feature from an n-th layer of the one neural network and extract a second deep feature from an m-th layer of the one neural network.
• the electronic apparatus 100 may store an extracted deep feature together with information about a neural network from which the deep feature is extracted, layer information of the neural network, and sub-network information.
• the electronic apparatus 100 may classify a plurality of images or search for an image by using deep features extracted from one neural network.
• FIG. 4 is a diagram illustrating an example result of classifying, by the electronic apparatus 100, a plurality of images using a feature extraction model and a classification model, which are trained based on general-purpose data, according to an example embodiment of the present disclosure.
• the electronic apparatus 100 may obtain a plurality of images. For example, as illustrated in FIG. 4, the electronic apparatus 100 obtain first through tenth images A1, A2, A3,A4, A5, A6, A7, A8, A9 and A10.
• the electronic apparatus 100 may classify the obtained first through tenth images A1 through A10 into certain groups using a feature extraction model and a classification model.
• the feature extraction model and the classification model may be models that are pre-trained based on general-purpose data.
• the feature extraction model and the classification model may be models trained to classify a plurality of images into six categories, e.g., 'person', 'dog', 'landscape', 'document', 'food', and 'road', but are not limited thereto.
• types and number of categories may be determined by the feature extraction model and the classification model via learning, or may be set based on a user input.
• the electronic apparatus 100 classifies the first through tenth images A1 through A10 using the feature extraction model and the classification model
• the first through fifth images A1 through A5, the seventh and eighth images A7 and A8, and the tenth image A10 may be classified into a 'dog' category
• the sixth and ninth images A6 and A9 may be classified into a 'person' category.
• first through tenth images A1 through A10 are classified according to the feature extraction model and the classification model, which are trained based on general-purpose data, only the 'dog' and 'person' categories are used, and 'landscape', 'food', 'road', and 'document' categories are not used.
• FIG. 5 is a diagram illustrating an example result of classifying, by the electronic apparatus 100, a plurality of images using a feature extraction model and a classifying mode, which are updated, according to an example embodiment of the present disclosure.
• the electronic apparatus 100 may train and update at least one of a feature extraction model and a classification model using user images. For example, the electronic apparatus 100 may determine necessity of updating the feature extraction model and the classification model using a result of classifying the first through tenth images A1 through A10 of FIG. 4. Also, the electronic apparatus 100 may train and update at least one of the feature extraction model and the classification model by using the user images.
• the electronic apparatus 100 may re-classify pre-classified images using the updated feature extraction model and the updated classification model.
• the first, second, and eighth images A1, A2, and A8 may be classified into a first group
• the third and fourth images A3 and A4 may be classified into a second group
• the sixth and ninth images A6 and A9 may be classified into a third group
• the fifth, seventh, and tenth images A5, A7, and A10 may be classified into a fourth group
• the sixth to eighth and tenth images A6 to A8 and A10 may be classified into a fifth group
• the first through fourth images A1 through A4, seventh image A7, and ninth image A9 may be classified into a sixth group.
• the first through tenth images A1 through A10 may be classified overlappingly into the first through sixth groups.
• the first through tenth images A1 through A10 are classified into two groups in FIG. 4, but are classified into six groups in FIG. 5.
• the feature extraction model and the classification model which are trained based on user data, may more variously classify user images.
• the electronic apparatus 100 may generate group names of groups into which a plurality of images are classified automatically or based on a user input.
• the electronic apparatus 100 may automatically generate the group names of the groups by using a language model.
• the electronic apparatus 100 may detect a keyword corresponding to an image by comparing similarity between keywords in a language model and similarity between deep features of images.
• a group name of the first group may be set to "Beagle”.
• a second keyword having "Beagle” and similarity (distance between deep features) between a deep feature of an image included in the first group and a deep feature of an image included in the second group, by using a language model, may be determined.
• the determined second keyword may be assigned as a group name of the second group.
• the electronic apparatus 100 may determine group names corresponding to the first through third groups based on information that the first through third groups are groups corresponding to a 'dog' in FIG. 4.
• the electronic apparatus 100 may determine a keyword corresponding to each of the deep feature of the image included in the first group, the deep feature of the image included in the second group, and the deep feature of the image included in the third group, by comparing similarity between lower keywords of "dog" in a language model and similarity between the deep features of the images included in the first through third groups.
• the electronic apparatus 100 may assign the determined keywords as the group names of the respective groups.
• the first group may have a group name "Beagle”
• the second group may have a group name "Poodle”
• the third group may have a group name "Anny”
• the fourth group may have a group name "Golden Retriever”
• the fifth group may have a group name "Indoor”
• the sixth group may have a group name "Outdoor”.
• the electronic apparatus 100 may generate a folder corresponding to each of the first through sixth groups, and store images classified as the same group in the same folder. Also, the electronic apparatus 100 may determine keywords respectively corresponding to images included in a group.
• images of the first user may include first through eleventh images B1, B2, B3, B4, B5, B6, B7, B8, B9, B10 and B11.
• the images of the first user may, for example, and without limitation, be images obtained by a first user device, and may be images stored in the first user device, images captured by the first user device, or images received from an external apparatus, or the like.
• the first user device may classify the first through eleventh images B1 through B11 into certain groups using a first feature extraction model and a first classification model.
• the first feature extraction model and the first classification model may be a feature extraction model and a classification model, which are trained using the images of the first user.
• the first feature extraction model extracts a deep feature of each of the images of the first user, and the first classification model may classify the images of the first user into certain groups based on the deep feature of each of the images of the first user.
• the type or number of the certain groups into which the images of the first user are classified may be determined by the first feature extraction model and the first classification model via user data-based learning, or may be set based on a user input.
• the first user device may generate group names of the first through sixth groups into which the first through eleventh images B1 through B11 are classified automatically or based on a user input. For example, the first user device may generate the group names or keywords automatically by using deep feature information and a language model of the first through eleventh images B1 through B11 included in the first through sixth groups.
• the electronic apparatus 100 provides a feature extraction model and a classification model, which are optimized according to users, and thus a plurality of images may be classified using classification criteria optimized according to users, instead of one classification criterion.
• operations S710 through S790 of FIG. 7A may each be performed by the electronic apparatus 100 or the server 200 according to an embodiment.
• operations S710 through S770 are performed by the server 200, but an embodiment is not limited thereto, and may be performed by the electronic apparatus 100.
• operations S730 through S750, and S790 are performed by the electronic apparatus 100, but an embodiment is not limited thereto, and may be performed by the server 200.
• FIG. 7B is a flowchart illustrating an example method of operating the server 200, a first processor 120a, and a second processor 120b, according to an example embodiment of the present disclosure.
• the electronic apparatus 100 may include the first processor 120a and the second processor 120b.
• the second processor 120b may, for example, be manufactured in a form of an AI exclusive hardware chip that performs an image classification function using a data recognition model (for example, a feature extraction model and a data classification model).
• a data recognition model for example, a feature extraction model and a data classification model.
• the AI exclusive hardware chip may include GPU for the data recognition model using visual understanding as an element technology.
• the server 200 may transmit the feature extraction model and the classification model to the electronic apparatus 100.
• the electronic apparatus 100 may be set such that the second processor 120b may use the feature extraction model and the classifying mode, in operation S7120.
• the first processor 120a may transmit the obtained images to the second processor 120b, in operation S7140.
• the second processor 120b may re-extract the deep features of the pre-classified plurality of images, and re-classify the plurality of images based on the re-extracted deep features.
• the electronic apparatus 100 may include the first processor 120a, the second processor 120b, and the third processor 120c.
• the second processor 120b may transmit the extracted deep features and the images to the third processor 120c, in operation S7260.
• the second processor 120b may transmit the plurality of images and the extracted deep features to the third processor 120c, in operation S7320.
• the electronic apparatus 100 may obtain a first image, in operation S810.
• the first image may be an image captured by or pre-stored in the electronic apparatus 100.
• the first image may be received from an external apparatus.
• the electronic apparatus 100 may extract an image similar to the first image based on similarity between the deep feature of the first image and deep features of the pre-stored images, by, for example, using a classification model. For example, the electronic apparatus 100 may extract, from the pre-stored images, an image of which a difference value from a vector of the deep feature of the first image is within a pre-set range.
• FIG. 8B is a flowchart illustrating an example method of operating the first processor 120a and the second processor 120b included in the electronic apparatus 100, according to an example embodiment of the present disclosure.
• the first processor 120a may obtain a first image in operation S8110.
• the second processor 120b may extract at least one image having a deep feature similar to the deep feature of the first image, from among pre-stored images, using a classification model and the deep feature of the first image, in operation S8140. For example, the second processor 120b may extract, from the pre-stored images, an image of which a difference value from a vector of the deep feature of the first image is within a pre-set range.
• the first processor 120a may obtain a first image in operation S8210.
• the third processor 120c may extract at least one image having a deep feature similar to the deep feature of the first image from among pre-stored images, using a classification model and the deep feature of the first image, in operation S8250.
• the second processor 120b may extract, from the pre-stored images, an image of which a difference value from a vector of the deep feature of the first image is within a pre-set range.
• the first processor 120a may display the extracted at least one image on a display in operation S8270.
• FIGS. 9 and 10 are diagrams illustrating an example method of searching for an image, the method performed by the electronic apparatus 100, according to an example embodiment of the present disclosure.
• the electronic apparatus 100 may obtain a first image 910.
• the first image 910 may be an image captured by or pre-stored in the electronic apparatus 100.
• the first image 910 may be an image received from an external apparatus.
• the electronic apparatus 100 may extract a deep feature of the first image 910 using a feature extraction model.
• the feature extraction model may include at least one neural network, and the deep feature may include a vector extracted from at least one layer by inputting the first image 910 into the at least one neural network.
• the electronic apparatus 100 may search for an image similar to the first image 910 based on similarity between the deep feature of the first image 910 and deep features of a pre-stored plurality of images 920, by using a classification model.
• the pre-stored plurality of images 920 may include deep features extracted from each of the images 920.
• a deep feature may be stored as EXIF information of each of a plurality of images.
• the deep feature of the first image 910 and the deep features of the plurality of images 920 may be deep features extracted using one neural network having one version.
• the deep feature of the first image 910 and the deep features of the plurality of images 920 may be deep features extracted from the same sub-network of one neural network, or from the same layer of one neural network.
• the electronic apparatus 100 may extract an image of which a difference value with the deep feature of the first image 910, from among the plurality of images 920, is within a pre-set range. For example, as illustrated in FIG. 9, from among the plurality of images 920, first through eighth images 930 may be extracted and displayed as images similar to the first image 910.
• the electronic apparatus 100 may only search stored images by using the keyword, and if the user stored a 'dog' image with a keyword (for example, happy) irrelevant to a 'dog', the electronic apparatus 100 is unable to find the 'dog' image stored in the keyword irrelevant to a 'dog'. Accordingly, the user has to remember a keyword stored with respect to an image to find the image.
• a keyword for example, dog, beagle, or puppy
• the electronic apparatus 100 extracts a deep feature of a 'dog' image using an image including a 'dog' instead of using a keyword, and search for images having a deep feature similar to the extracted deep feature, and thus the user may find a desired image without having to remember all keywords.
• the electronic apparatus 100 may obtain a second image 1010.
• the electronic apparatus 100 may extract a deep feature of the second image 1010 using a feature extraction model.
• the deep feature of the second image 1010 may indicate a feature of a bridge, a feature of a night view, a feature of a river, but is not limited thereto.
• the electronic apparatus 100 may search for an image similar to the second image 1010 based on similarity between the deep feature of the second image 1010 and deep features of the pre-stored plurality of images 920, by using a classification model. For example, the electronic apparatus 100 may extract and display, from among the plurality of images 920, images 1030 of which a difference value with the deep feature of the second image 1010 is within a pre-set range, as images similar to the second image 1010.
• the user has to input related keywords (bridge and night view) so as to search for a 'night view with bridge' image.
• the electronic apparatus 100 extracts a deep feature of the 'night view with bridge' image by using the 'night view with bridge' image instead of using a keyword, and search for images having a deep feature similar to the extracted deep feature, and thus the user may find a desired image without having to remember all keywords.
• the electronic apparatus 100 may perform an image search through a keyword input. For example, when a keyword is input, the electronic apparatus 100 may determine a keyword similar to the input keyword by using a language model, and search for images corresponding to the similar keyword. When a tag name for a first image corresponds to the similar keyword, the electronic apparatus 100 may extract the first image and display the first image as an image search result. Also, when a group name of images corresponds to the similar keyword, the electronic apparatus 100 may extract second images included in a group of the group name and display the second images as image search results.
• an embodiment is not limited thereto.
• FIG. 11 is a block diagram illustrating an example configuration of the electronic apparatus 100, according to an example embodiment of the present disclosure.
• the electronic apparatus 100 may include a processor (e.g., including processing circuitry) 120, a display 140, and a memory 130.
• a processor e.g., including processing circuitry
• the display 140 generates a driving signal by converting an image signal, a data signal, an on-screen display (OSD) signal, or a control signal processed by the processor 120.
• the display 140 may be embodied as a plasma display panel (PDP), a liquid crystal display (PDP), a organic light-emitting display (OLED), or a flexible display, or may be embodied as a 3-dimensional (3D) display, or the like, but is not limited thereto.
• the display 140 may be configured as a touch screen to be used as an input device as well as an output device.
• the display 140 may display an image.
• the image displayed on the display 140 may be at least one of an image captured by the electronic apparatus 100, an image stored in the electronic apparatus 100, or an image received from an external apparatus.
• an embodiment is not limited thereto.
• the processor 120 may include various processing circuitry and execute at least one program stored in the memory 130.
• the processor 120 may, for example, and without limitation, include a single core, a dual core, a triple core, a quad core, or a multiple core.
• the processor 120 may include a plurality of processors.
• the processor 120 may include a main processor (not shown) and a sub-processor (not shown) that operates in a sleep mode.
• the memory 130 may store various types of data, programs, or applications for driving and controlling the electronic apparatus 100.
• the program stored in the memory 130 may include at least one instruction.
• the program (at least one instruction) or the application stored in the memory 130 may be executed by the processor 120.
• the processor 120 may execute the at least one instruction stored in the memory 130 to obtain a plurality of images and extract deep features with respect to the plurality of images using a feature extraction model.
• the feature extraction model may include a first neural network, and the processor 120 may extract a vector included in the deep feature from at least one layer of the first neural network by inputting each of the plurality of images into the first neural network.
• the processor 120 may classify the plurality of images into certain groups using the extracted deep features and a classification model.
• the classification model may include a second neural network that classifies the plurality of images into the certain groups based on similarity between the deep features of the plurality of images.
• the processor 120 may execute the at least one instruction stored in the memory 130 to store the classified plurality of images together with the deep features. Also, the processor 120 may control the display 140 to display a result of the classifying, determine necessity of updating the feature extraction model and the classification model by using the result of the classifying, and train and update at least one of the feature extraction model and the classification model based on a result of the determining. When at least one of the feature extraction model and the classification model is updated, the processor 120 may re-classify the pre-classified plurality of images by using the updated feature extraction model and classification model.
• the processor 120 may extract a deep feature with respect to a first image, and extract at least one image having a deep feature similar to the deep feature of the first image from among the classified plurality of images.
• FIG. 12 is a block diagram illustrating the processor 120 according to an example embodiment of the present disclosure.
• the processor 120 may include a data learning unit (e.g., including processing circuitry and/or program elements) 1300 and a data classifying unit (e.g., including processing circuitry and/or program elements) 1400.
• a data learning unit e.g., including processing circuitry and/or program elements
• a data classifying unit e.g., including processing circuitry and/or program elements
• the data learning unit 1300 may include processing circuitry and/or program elements configured to learn a determination criterion for classifying an image into a certain group.
• the data learning unit 1300 may learn a determination criterion about which data is to be used to classify an image into a certain group and how to classify an image by using data.
• the data learning unit 1300 may obtain data to be used for learning, and learn a determination criterion for classifying an image by applying the data to a data classification model described later.
• the data classification model according to an embodiment may include a feature extraction model and a classification model.
• the data classifying unit 1400 may classify an image based on the data.
• the data classifying unit 1400 may classify a plurality of images into certain groups using the learned data classification model.
• the data classifying unit 1400 may obtain certain data according to standards pre-set via learning, and classify an image based on the certain data using the data classification model in which the obtained certain data is used as an input value. Also, a result value output by the data classification model in which the obtained certain data is used as an input value may be used to update the data classification model.
• the data learning unit 1300 and the data classifying unit 1400 may be manufactured in a form of at least one hardware chip and included in an electronic apparatus.
• at least one of the data learning unit 1300 and the data classifying unit 1400 may be manufactured in a form of an artificial intelligence (AI)-exclusive hardware chip or may be manufactured as part of an existing general-purpose processor (for example, a central processing unit (CPU) or an application processor) or a graphic-exclusive processor (for example, a graphical processing unit (GPU)), and included in any electronic apparatus.
• AI artificial intelligence
• the data learning unit 1300 and the data classifying unit 1400 may be included in one electronic apparatus or in different electronic apparatuses.
• one of the data learning unit 1300 and the data classifying unit 1400 may be included in an electronic apparatus and the other one may be included in a server.
• model information built by the data learning unit 1300 may be provided to the data classifying unit 1400 or data input to the data classifying unit 1400 may be provided to the data learning unit 1300 as additional learning data, via wires or wirelessly.
• At least one of the data learning unit 1300 and the data classifying unit 1400 may be implemented in a software module.
• the software module may be stored in a non-transitory computer-readable recording medium.
• the software module may be provided by an OS or a certain application. Alternatively, a part of the software module may be provided by an OS and the remainder of the software module may be provided by a certain application.
• FIG. 13 is a block diagram illustrating an example data learning unit 1300 according to an example embodiment of the present disclosure.
• the data learning unit 1300 may include a data obtainer (e.g., including processing circuitry and/or program elements) 1310, a pre-processor (e.g., including processing circuitry and/or program elements) 1320, a learning data selector (e.g., including processing circuitry and/or program elements) 1330, a model learner (e.g., including processing circuitry and/or program elements) 1340, and a model evaluator (e.g., including processing circuitry and/or program elements) 1350, but is not limited thereto.
• the data learning unit 1300 may include some of the above components.
• the data learning unit 1300 may only include the data obtainer 1310 and the model learner 1340.
• the data learning unit 1300 may further include a component other than the above components.
• the data obtainer 1310 may obtain data required to classify a plurality of images into certain groups.
• the data obtainer 1310 may obtain data required to learn to classify an image.
• the data obtainer 1310 may obtain a plurality of pieces of image data.
• the data obtainer 1310 may receive image data through a camera of an electronic apparatus including the data learning unit 1300.
• the data obtainer 1310 may receive image data through an external apparatus communicable with the electronic apparatus including the data learning unit 1300.
• the pre-processor 1320 may pre-process the obtained data such that the obtained data may be used during the learning for classifying an image.
• the pre-processor 1320 may process the obtained data to a pre-set format such that the obtained data is used by the model train 1340 described later.
• the learning data selector 1330 may select data required for learning from the pre-processed data.
• the selected data may be provided to the model learner 1340.
• the learning data selector 1330 may select data required for learning from the pre-processed data according to a pre-set criterion for classifying an image. Also, the learning data selector 1330 may select data according to a criterion pre-set via learning of the model learner 1340 described later.
• the model learner 1340 may learn a determination criterion about how to classify images based on learning data. Also, the model learner 1340 may learn a selection criterion about which learning data is to be used to classify an image.
• the model learner 1340 may learn a determination criterion of extracting deep features of a plurality of images, and learn to classify the plurality of images into certain groups based on similarity between the deep features of the plurality of images.
• similarity between deep features may be indicated by a distance between vectors extracted from the deep features, wherein the similarity is high when the distance between the vectors is short, and the similarity is low when the distance between the vectors is long.
• a plurality of images of which a distance between vectors is within a pre-set range may be classified into one group.
• the model learner 1340 may train a data classification model for classifying a plurality of images, by using learning data.
• the data classification model may be a pre-built model.
• the data classification model may be pre-built by receiving basic learning data (for example, a sample image).
• the data classification model may be built considering an application field of the data application model, a learning purpose, or a computer performance of an electronic apparatus.
• the data classification model may be, for example, a neural network-based model.
• DNN, CNN, RNN, or BRDNN, or the like may be used as the data classification model, but an embodiment is not limited thereto.
• the model learner 1340 may determine, as a data classification model to be learned, a data classification model having high relevance between input learning data and basic learning data.
• the basic learning data may be pre-classified according to data types, and the data classification models may be pre-built according to data types.
• the basic learning data may be pre-classified based on various standards, such as a region where the basic learning data is generated, a time when the basic learning data is generated, a size of the basic learning data, a genre of the basic learning data, a generator of the basic learning data, and a type of an object in the basic learning data.
• model learner 1340 may train, for example, the data classification model using a training algorithm including error back-propagation or gradient descent.
• the model learner 1340 may store the trained data classification model.
• the model learner 1340 may store the data classification model in a memory of an electronic apparatus including the data classifying unit 1400.
• the model learner 1340 may store the trained data classification model in a memory of an electronic apparatus including the data classifying unit 1400 to be described later.
• the model learner 1340 may store the trained data classification model in a memory of a server connected to the electronic apparatus via a wired network or a wireless network.
• the memory in which the trained data classification model is stored may also store, for example, an instruction or data related to another at least one component of the electronic apparatus.
• the memory may store software and/or a program.
• the program may include, for example, a kernel, middleware, an application programming interface (API) and/or an application program (or "application").
• the model evaluator 1350 may input evaluation data into the data classification model, and when recognition results output from the evaluation data does not satisfy a certain standard, enable the model learner 1340 to train the data classification model again.
• the evaluation data may be data pre-set to evaluate the data classification model.
• the model evaluator 1350 may determine that the recognition results does not satisfy the certain standard when the number or proportion of pieces of evaluation data of which classification results are not accurate exceeds a pre-set threshold value, from among classification results of the trained data classification model with respect to the evaluation data. For example, when the certain standard is 2% and the trained data recognition model outputs wrong recognition results with respect to over 20 pieces of evaluation data from among 1000 pieces of evaluation data, the model evaluator 1350 may determine that the trained data classification model is not suitable.
• At least one of the data obtainer 1310, the pre-processor 1320, the learning data selector 1330, the model learner 1340, and the model evaluator 1350 in the data learning unit 1300 may be manufactured in at least one hardware chip form and included in an electronic apparatus.
• at least one of the data obtainer 1310, the pre-processor 1320, the learning data selector 1330, the model learner 1340, and the model evaluator 1350 may be manufactured in an AI-exclusive hardware chip or may be manufactured as part of an existing general-purpose processor (for example, CPU or an application processor) or a graphic-exclusive processor (for example, GPU), and included in any electronic apparatus described above.
• At least one of the data obtainer 1310, the pre-processor 1320, the learning data selector 1330, the model learner 1340, and the model evaluator 1350 may be implemented in a software module.
• the software module may be stored in a non-transitory computer-readable recording medium.
• the software module may be provided by an OS or a certain application. Alternatively, a part of the software module may be provided by an OS and the remainder of the software module may be provided by a certain application.
• FIG. 14 is a block diagram illustrating an example data classifying unit 1400 according to an example embodiment of the present disclosure.
• the data classifying unit 1400 may include a data obtainer (e.g., including processing circuitry and/or program elements) 1410, a pre-processor (e.g., including processing circuitry and/or program elements) 1420, a classification data selector (e.g., including processing circuitry and/or program elements) 1430, a classification result provider (e.g., including processing circuitry and/or program elements) 1440, and a model updater (e.g., including processing circuitry and/or program elements) 1450.
• a data obtainer e.g., including processing circuitry and/or program elements
• a pre-processor e.g., including processing circuitry and/or program elements
• a classification data selector e.g., including processing circuitry and/or program elements
• a classification result provider e.g., including processing circuitry and/or program elements
• a model updater e.g., including processing circuitry and/or program elements
• the data obtainer 1410 may obtain data required for image classification, and the pre-processor 1420 may pre-process the obtained data such that the obtained data is used for image classification.
• the pre-processor 1420 may process the obtained data to a pre-set format such that the obtained data is used for image classification.
• the classification data selector 1430 may select data required for image classification from the pre-processed data.
• the selected data may be provided to the classification result provider 1440.
• the classification data selector 1430 may select some or all of pieces of the pre-processed data based on a pre-set criterion for image classification. Also, the classification data selector 1430 may select data according to a criterion pre-set via learning by the model learner 1340.
• the classification result provider 1440 may classify an image by applying the selected data to a data classification model.
• the classification result provider 1440 may provide a classification result according to a classification purpose of the data.
• the classification result provider 1440 may apply the selected data to the data classification model using, as an input value, the data selected by the classification data selector 1430. Also, the classification result may be determined by the data classification model.
• the classification result provider 1440 may provide a result of classifying a plurality of images into certain groups. Images that are classified into one group may be stored in the same folder.
• the classification result provider 1440 may estimate an image similar to another image.
• the classification result provider 1440 may estimate an image having a deep feature similar to that of a first image (for example, one of a pre-stored images or newly input images).
• the model updater 1450 may update the data classification model based on evaluation on the classification result provided by the classification result provider 1440. For example, the model updater 1450 may determine whether the data classification model needs to be updated according to the classification result provided by the classification result provider 1440, and when the data classification model needs to be updated, update the data classification model using the model learner 1340. The model learner 1340 may re-train the data classification model by using image data of a user to update the data classification model.
• At least one of the data obtainer 1410, the pre-processor 1420, the classification data selector 1430, the classification result provider 1440, and the model updater 1450 in the data classifying unit 1400 may be manufactured in at least one hardware chip form and included in an electronic apparatus.
• at least one of the data obtainer 1410, the pre-processor 1420, the classification data selector 1430, the classification result provider 1440, and the model updater 1450 may be manufactured in an AI-exclusive hardware chip or may be manufactured as part of an existing general-purpose processor (for example, CPU or an application processor) or a graphic-exclusive processor (for example, GPU), and included in any electronic apparatus described above.
• the data obtainer 1410, the pre-processor 1420, the classification data selector 1430, the classification result provider 1440, and the model updater 1450 may be included in one electronic apparatus or on different electronic apparatuses.
• some of the data obtainer 1410, the pre-processor 1420, the classification data selector 1430, the classification result provider 1440, and the model updater 1450 may be included in an electronic apparatus, and the remainder may be included in a server.
• At least one of the data obtainer 1410, the pre-processor 1420, the classification data selector 1430, the classification result provider 1440, and the model updater 1450 may be implemented in a software module.
• the software module may be stored in a non-transitory computer-readable recording medium.
• the software module may be provided by an OS or a certain application. Alternatively, a part of the software module may be provided by an OS and the remainder of the software module may be provided by a certain application.
• FIG. 15 is a diagram illustrating an example of an electronic apparatus 1000 and a server 2000 interworking together to learn and recognize data, according to an embodiment.
• the server 2000 may analyze a user image to learn a criterion for classifying an image, and the electronic apparatus 100 may classify a plurality of images based on a learning result by the server 2000.
• the server 2000 may include a data obtainer (e.g., including processing circuitry and/or program elements) 2310, a pre-processor (e.g., including processing circuitry and/or program elements) 2320, a learning data selector (e.g., including processing circuitry and/or program elements) 2330, a model learner (e.g., including processing circuitry and/or program elements) 2340 and a model evaluator (e.g., including processing circuitry and/or program elements) 2350.
• a data obtainer e.g., including processing circuitry and/or program elements
• a pre-processor e.g., including processing circuitry and/or program elements
• a learning data selector e.g., including processing circuitry and/or program elements
• a model learner 2340 of the server 200 may perform a function of the model learner 1340 of FIG. 13.
• the model learner 2340 may learn a criterion of extracting deep features of a plurality of images, and learn to classify the plurality of images into certain groups based on similarity between the deep features of the plurality of images.
• the model learner 2340 may obtain data to be used for learning, and apply the obtained data to a data classification model to learn a criterion for classifying the plurality of images.
• the classification result provider 1440 of the electronic apparatus 1000 may classify the plurality of images by applying data selected by the classification data selector 1430 to the data classification model generated by the server 2000. For example, the classification result provider 1440 may transmit the data selected by the classification data selector 1430 to the server 2000, and request the server 2000 to classify the plurality of images by applying the data selected by the classification data selector 1430 to the data classification model. Also, the classification result provider 1440 may provide a result of classifying the plurality of images into certain groups. Images classified into one group may be stored in the same folder.
• FIG. 16 is a block diagram illustrating an example configuration of an electronic apparatus 300, according to another example embodiment of the present disclosure.
• the electronic apparatus 300 of FIG. 16 may be an example of the electronic apparatus 100 of FIG. 1.
• the controller 330 of FIG. 16 may correspond to the processor 120 of FIG. 11, the storage unit 380 of FIG. 16 may correspond to the memory 130 of FIG. 11, and a display 351 of FIG. 16 may correspond to the display 140 of FIG. 11. Accordingly, details of FIG. 16 that are the same as those of FIG. 11 are not repeated here.
• the short-range wireless communication unit 341 may include various short-range wireless communication circuitry, such as, for example, and without limitation, a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a near-field communication (NFC) unit, a wireless local area network (WLAN) (Wi-Fi) communication unit, a Zigbee communication unit, an Infrared Data Association (IrDA) communication unit, a Wi-Fi direct (WFD) communication unit, an ultra-wideband (UWB) communication unit, and an Ant+ communication unit, but is not limited thereto.
• BLE Bluetooth Low Energy
• NFC near-field communication
• Wi-Fi wireless local area network
• Zigbee communication unit Zigbee communication unit
• IrDA Infrared Data Association
• WFD Wi-Fi direct
• UWB ultra-wideband
• the broadcast receiving unit 343 may include various broadcast receiving circuitry and receives a broadcast signal and/or broadcast related information from an external source, through a broadcast channel.
• the broadcast channel may include a satellite channel or a terrestrial broadcasting channel.
• the electronic apparatus 300 may not include the broadcast receiving unit 343.
• the vibration motor 353 may output a vibration signal.
• the vibration motor 353 may output a vibration signal corresponding to an output of audio data or video data.
• the vibration motor 353 may output a vibration signal when a touch screen is touched.
• the notification module 383 may generate a signal for notifying event generation in the electronic apparatus 300. Examples of an event generated by the electronic apparatus 300 include call signal reception, message reception, key signal input, and schedule notification.
• the notification module 383 may output a notification signal in a video signal format through the display 351, in an audio signal format through the sound output unit 352, or in a vibration signal format through the vibration motor 353.
• the electronic apparatuses 100 and 300 illustrated in FIG. 11 and 16 are only examples, and components of the electronic apparatus 100 or 300 may be combined, or an element may be added to or omitted from the electronic apparatus 100 or 300, according to an embodiment.
• at least two components may be combined into one component, or one component may be divided into at least two components.
• functions performed by the components are only examples, and detailed operations do not limit the scope of the present disclosure.
• the computer program product may include a software program and a computer-readable recording medium having recorded thereon the software program.
• the computer program product may include a product (for example, a downloadable application) in a form of a software program electronically distributed through an electronic market (for example, Google Play or Appstore) or a manufacturer of a patient monitoring apparatus.
• an electronic market for example, Google Play or Appstore
• the storage medium may be a storage medium of a server of the manufacturer, a server of the electronic market, or a relay server that temporarily stores the software program.
• the third apparatus may execute the computer program product to control the electronic apparatus connected to the third apparatus to perform a method according to an embodiment.
• the third apparatus may download the computer program product from the server and execute the downloaded computer program product.
• the third apparatus may execute the computer program product that is pre-loaded to perform a method according to an embodiment.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Evolutionary Computation (AREA)
• Artificial Intelligence (AREA)
• General Engineering & Computer Science (AREA)
• Life Sciences & Earth Sciences (AREA)
• Software Systems (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Multimedia (AREA)
• Molecular Biology (AREA)
• Computing Systems (AREA)
• Biomedical Technology (AREA)
• Databases & Information Systems (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Evolutionary Biology (AREA)
• Bioinformatics & Computational Biology (AREA)
• Mathematical Physics (AREA)
• Computational Linguistics (AREA)
• Biophysics (AREA)
• Library & Information Science (AREA)
• Biodiversity & Conservation Biology (AREA)
• Medical Informatics (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (3)

Publications (2)

Family

ID=62917879

Family Applications (1)

Country Status (3)

Cited By (3)

Families Citing this family (15)

Family Cites Families (5)

• 2017
  • 2017-10-20 KR KR1020170136612A patent/KR102428920B1/ko active IP Right Grant
• 2018
  • 2018-01-03 CN CN201880005869.1A patent/CN110168530B/zh active Active
  • 2018-01-03 EP EP18735947.6A patent/EP3545436A4/de active Pending

Also Published As

Similar Documents

Legal Events