KR20180074562A - Electronic device and Method for controlling the electronic device thereof - Google Patents

Abstract

The present invention provides an electronic device worn on the head of a user to provide an image, and a control method thereof. The control method of an electronic device learns content features in accordance with reaction features of the eyes of a user wearing the electronic device to construct a user model, and processes content played based on the reaction features of the eyes of the user corresponding to detected content features if the content features stored in the user model are detected from the played content while the content is played. The electronic device can estimate features of content by using rule-based or artificial intelligence algorithm. If the features of the content are estimated by using the artificial intelligence algorithm, an image processing device can estimate the features of the content by using machine-learning, neural-network or deep-learning algorithm.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an electronic device and a control method thereof,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an electronic device and a control method thereof, and more particularly to an electronic device capable of controlling reproduction of content reproduced according to a content characteristic by learning content characteristics according to a reaction characteristic of a user's eye, .

The present disclosure also relates to an artificial intelligence (AI) system and its application technology, which simulate functions such as recognition and judgment of a human brain using a machine learning algorithm such as deep learning.

Recently, attention has been paid to an electronic device (for example, an HMD (Head Mounted Device)) that provides a virtual reality worn by a user. For example, when an HMD is worn, In addition, I can enjoy the realistic game with the 360-degree view of the existing 2D-based game, and many VR contents are being released starting from the game contents, and also, It is expected to be used in various fields such as distance learning and medical services.

In the case of such an HMD, the distance between the user's eyes and the electronic device is so close that the visual stimulus applied to the user's eyes may be very high depending on a specific color or a specific brightness change. As a result, not only the fatigue of the user's eyes can be increased, but also the possibility of causing disease in the user's eyes is caused.

In addition, when a user has a fear of a specific object (e.g., knife, sharp object), if a specific object is displayed on the image provided by the HMD, the user can maximize the fear and watch the image using the HMD You will feel uncomfortable.

In recent years, artificial intelligence systems have also been introduced in the field of image processing.

An artificial intelligence system is a computer system that implements human-level intelligence, and unlike existing rule-based smart systems, a machine learns, judges, and clears itself. Artificial intelligence systems improve recognition rate and understand user's taste more accurately As a result, existing Rule-based smart systems are increasingly being replaced by deep-run-based artificial intelligence systems.

Artificial intelligence technology consists of element technologies that utilize deep learning and machine learning.

Machine learning is an algorithm technology that classifies / learns the characteristics of input data by itself. Element technology is a technology that simulates functions such as recognition and judgment of human brain using machine learning algorithms such as deep learning. Understanding, reasoning / prediction, knowledge representation, and motion control.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing, applying, and processing human language / characters, including natural language processing, machine translation, dialogue system, query response, speech recognition / synthesis, and the like. Visual understanding is a technology that recognizes and processes objects as human vision. It includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, and image enhancement. Inference prediction is a technique for judging and logically inferring and predicting information, including knowledge / probability based reasoning, optimization prediction, preference base planning, and recommendation. Knowledge representation is a technology for automating human experience information into knowledge data, including knowledge building (data generation / classification) and knowledge management (data utilization). The motion control is a technique for controlling the autonomous travel of the vehicle and the motion of the robot, and includes motion control (navigation, collision, traveling), operation control (behavior control), and the like.

It is an object of the present invention to provide an electronic device for learning a content feature according to a reaction characteristic of a user's eye to construct a user model, and to reproduce content using the established user model and a control method thereof.

According to another aspect of the present invention, there is provided a method of controlling an electronic device worn on a user's head, comprising: learning a content feature according to a reaction characteristic of a user's eyes wearing the electronic device, ; And processing the content to be played back based on a response feature of the user's eyes corresponding to the detected content feature, when a content feature stored in the user model is detected from the content to be played back while the content is being played back .

The electronic device worn on the head of the user includes a photographing section for photographing the user's eyes; A memory for storing a user model constructed by learning content characteristics according to a reaction characteristic of a user's eye wearing the electronic device; And a processor for processing the content to be played back based on a reaction characteristic of the user's eye corresponding to the detected content feature, when a content feature stored in the user model is detected from the content to be played back while the content is being played back .

According to various embodiments of the present invention as described above, the user can view the video image more comfortably and without irritation by performing the image processing on the content feature that the user has reacted negatively. Further, by observing disease information on the user's eyes, information on the abnormal state of the user's eyes can be provided.

BRIEF DESCRIPTION OF THE DRAWINGS Figures 1A and 1B are block diagrams, schematically illustrating the configuration of an electronic device, in accordance with one embodiment of the present disclosure;
Figure 2 is a block diagram detailing the configuration of an electronic device, in accordance with one embodiment of the present disclosure;
3 is a block diagram illustrating a configuration of a processor included in an electronic device, according to one embodiment of the present disclosure;
4A is a block diagram showing a configuration of a data learning unit and a canant control unit included in an electronic device according to an embodiment of the present disclosure;
4B is a diagram illustrating an example of learning and recognizing data by interlocking with an electronic device and a server, according to one embodiment of the present disclosure;
5 is a diagram for explaining a data learning unit according to an embodiment of the present disclosure;
6 is a diagram for explaining a method of processing content using a content control unit according to an embodiment of the present disclosure;
FIGS. 7A and 7B are views showing UIs for modifying the reaction characteristics and content characteristics of the user's eye, according to one embodiment of the present disclosure;
8A and 8B are diagrams for explaining an embodiment in which a user performs image processing for an object having a negative emotion, according to an embodiment of the present disclosure,
9 is a diagram illustrating an embodiment for determining disease information for a user's eye, according to one embodiment of the present disclosure;
10 illustrates a UI for guiding disease information to a user's eye, according to one embodiment of the present disclosure;
11 is a flowchart for explaining a control method of an electronic device according to an embodiment of the present disclosure;
12 is a diagram for explaining a system including a portable terminal and an electronic device according to an embodiment of the present disclosure;
13 is a flowchart for explaining a situation of estimating a content characteristic when an electronic device according to an embodiment includes a first processor and a second processor.
14 is a flowchart for explaining a situation in which an electronic device according to an embodiment estimates a content characteristic using a server.
15 is a flowchart for explaining a situation in which an electronic device according to an embodiment estimates an abnormal state related to eyes using a server.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear. And the terms used below are terms defined in consideration of the functions in this disclosure, which may vary depending on the user, operator or custom. Therefore, the definition should be based on the contents throughout this specification.

Terms including ordinals such as first, second, etc. may be used to describe various elements, but the elements are not limited by terms. Terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. The term " and / or " includes any combination of a plurality of related items or any of a plurality of related items.

The terminology used herein is used to describe the embodiments and is not intended to limit and / or limit the disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprising", "having", or the like are intended to specify that there are stated features, numbers, operations, acts, elements, parts or combinations thereof, , But do not preclude the presence or addition of one or more other features, elements, components, components, or combinations thereof.

In the embodiment, 'module' or 'sub' performs at least one function or operation, and may be implemented in hardware or software, or in a combination of hardware or software. In addition, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module except for 'module' or 'module' which needs to be implemented by specific hardware, and may be implemented by at least one processor.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings. FIG. 1A is a block diagram that schematically illustrates the configuration of an electronic device 100, in accordance with one embodiment of the present disclosure. 1A, an electronic device 100 includes a display 110, a photographing unit 120, a processor 130, and a memory 140. [ In this case, the electronic device 100 may be a head-up display device that can be worn on a user's head to provide an image as shown in FIG. 1B, but this is merely an example, Other electronic devices (e.g., smart phones, tablet PCs, notebook PCs, etc.).

The display 110 displays an image. In particular, the display 110 may display images obtained from various sources in 3D form. At this time, the display 110 may be positioned on the inner surface of the electronic device 100 viewed by the user's eyes when the user wears the electronic device 100. Therefore, the display unit 110 is more immersive to the user who wears the electronic device 100, and can provide stereoscopic images.

The photographing unit 120 photographs the user's eyes. In particular, the photographing unit 120 can be located on the inner side of the electronic device 100 to photograph the user's eyes.

The memory 140 stores various data and programs for controlling the electronic device 100. In particular, the memory 140 may store the user model constructed by learning the content characteristics according to the reaction characteristics of the user's eyes photographed by the photographing unit 120. [

The processor 130 may control the overall operation of the electronic device 100. When the content feature stored in the user model is detected from the content reproduced during reproduction of the content, the processor 130 may process the content to be reproduced based on the reaction feature of the user's eyes corresponding to the detected content feature.

Specifically, the processor 130 can construct the user model by learning the content characteristics according to the reaction characteristics of the user's eyes photographed by the photographing unit 120. [ At this time, the reaction characteristic of the user's eye may be a feature of the shape, size, color change of the user's eye, or the reaction rate of the eye, and may include blinking of eyes, eye strain, eye grating, pupil dilation, pupil contraction and the like.

More specifically, the processor 130 may photograph an image including the eyes of the user wearing the electronic device 100 through the photographing unit 120 while the learning contents are reproduced. Then, when predetermined reaction characteristics of the user eyes included in the photographed image are detected. The processor 130 may acquire a content feature included in a content frame within a preset interval from a point in time when a predetermined response feature is sensed. The processor 130 can learn a predetermined response characteristic and a content characteristic to construct a user model. At this time, the content feature may include at least one of an object (e.g., a knife, a ghost) included in a content frame within a preset interval from the time when the predetermined response feature is sensed, a brightness change of the content frame, . Then, the constructed user model can store the predetermined response characteristic corresponding to the content characteristic.

The processor 130 may perform image processing on the content based on the constructed user model.

Specifically, the processor 130 may analyze the content to be reproduced during reproduction of the content to determine whether the content feature stored in the user model exists. When the first content feature stored in the user model exists in the content to be reproduced, the processor 130 determines whether the first content feature exists in the content frame including the first content feature based on the first content feature and the response feature corresponding to the first content feature It is possible to perform image processing on the image.

In an embodiment of the present invention, when the first content feature is a specific object and the response characteristic for a particular object is a negative response (e.g., blinking, blinking, blinking, etc.) A filtering process may be performed on the embedded object or a smoothing process may be performed. In another embodiment, if the first content feature is a specific brightness change and the user response to a particular brightness change is negative, the processor 130 may adjust the amount of change in a particular brightness change. In another embodiment, if the first content feature is a specific color change and the user response to a particular color change is negative, the processor 130 may adjust the saturation value of the particular color change.

On the other hand, if it is determined that the user response to the second content feature in the learned user model is a positive response, the processor 130 obtains a keyword for the content feature, and based on the obtained keyword, And control the display 110 to provide a list containing the determined recommended content.

In addition, the processor 130 accumulates the photographed images including the user's eyes in the memory 140, and analyzes the images included in the stored user's eyes to determine an abnormal state of the user's eyes. If it is determined that the user's eyes are abnormal, the processor 130 may control the display 110 to provide information on the abnormal state.

2 is a block diagram illustrating in detail the configuration of an electronic device 100, in accordance with one embodiment of the present disclosure. 2, the display 110, the photographing unit 120, the memory 140, the communication unit 150, the image processing unit 160, the audio output unit 170, the input unit 180, 190 and a processor 130.

The display 110 may display various image contents, information, UI, and the like provided by the electronic device 100. In particular, the display 110 may display various execution screens of the navigation application.

In particular, the display 110 may display images obtained from various sources in 3D form. Specifically, the display 110 displays the left eye image through the display corresponding to the left eye of the user, and displays the right eye image through the display corresponding to the right eye of the user to display the 3D image.

At this time, the display 110 may be positioned on the inner surface of the electronic device 100 viewed by the user's eyes when the user wears the electronic device 100. Therefore, the display unit 110 is more immersive to the user who wears the electronic device 100, and can provide stereoscopic images.

The photographing unit 120 may be located on the inner side of the electronic device 100 to photograph the user's eyes when the electronic device 100 worn the electronic device 100. [ At this time, the photographing unit 120 may be a general camera, but it may be implemented by an infrared camera or the like, which is only an example.

The memory 140 may store various programs and data necessary for the operation of the electronic device 100. The memory 140 may be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 140 is accessed by the processor 130 and read / write / modify / delete / update of the data by the processor 130 can be performed. The term memory in this disclosure includes memory 140, a memory card (not shown) (e.g., micro SD (not shown)) mounted in ROM 130 Card, memory stick).

In addition, the memory 140 may store programs and data for configuring various screens to be displayed in the display area of the display 110. The memory 140 may also store a user model that stores content features corresponding to the user's eye reaction characteristics, in accordance with one embodiment of the present disclosure.

The communication unit 150 can perform communication with various types of external devices according to various types of communication methods. The communication unit 150 may include at least one of a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, and an NFC chip. The processor 130 can communicate with an external server or various external devices using the communication unit 150. [

In particular, the communication unit 150 can communicate with an external electronic device (e.g., a smart phone, a navigation device, a server, and the like). For example, the communication unit 150 can receive image content from an external electronic device.

The image processing unit 160 performs image processing on the image data received from various sources. The image processing unit 160 may perform various image processes such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on image data.

The audio output unit 170 can output various kinds of audio data processed by the audio processing module, as well as various kinds of sounds and voice messages.

The input unit 180 may input a user command for operating the electronic device 100 and may transmit information about the user command to the processor 130. [ In particular, the input unit 180 may be a button (e.g., a physical button, an optical key, or a keypad) provided in the electronic device 100, but this is merely an embodiment, .

For example, the input unit 180 may be implemented as a touch panel for sensing a user's touch, a (digital) pen sensor, or the like. The touch panel may use at least one of electrostatic, pressure sensitive, infrared, and ultrasonic methods, for example. Further, the touch panel may further include a control circuit. The touch panel may further include a tactile layer to provide a tactile response to the user. The (digital) pen sensor may be part of, for example, a touch panel, or may include a separate summary sheet.

However, it is only one embodiment that the input unit 120 is implemented by a touch panel, a pen sensor, or the like, and may be implemented by various electronic devices such as a microphone for inputting user's utterance, a camera for capturing user's motion, a pointing device, .

The sensing unit 190 includes various sensors for sensing the state of the electronic device 100. [ In particular, the sensing unit 190 may include a GPS sensor for sensing the position of the electronic device 100, a motion sensor (e.g., an acceleration sensor, a gyro sensor, an electromagnetic sensor, etc.) , A camera sensor for photographing a route on which the car travels, and the like.

The processor 130 is electrically connected to various configurations of the electronic device 100 (e.g., the display 110, the memory 140, and the imaging unit 120) Can be controlled. In particular, the processor 130 may control the overall operation of the electronic device 100 using various programs stored in the memory 140.

3, the processor 130 includes a data learning unit 131 for generating a user model 300, a content control unit 132 for processing content using the user model 300, .

The data learning unit 131 can learn the user's eye reaction characteristic on the content characteristic. In particular, the data learning unit 131 can learn the feature of the content that the user's eyes react to based on the image of the user's eyes and the learning content. At this time, the data learning unit 131 can build a user model 300 that stores content characteristics according to the reaction characteristics of the user's eyes by learning whether the user shows specific reaction characteristics to specific content features.

The content control unit 132 processes the content to be played back based on the user model 300 generated by the data learning unit 131. [ Specifically, when the content feature stored in the user model 300 is detected, the content control unit 132 can perform image processing on the content based on the reaction feature of the user's eyes corresponding to the detected content feature.

At least one of the data learning unit 131 and the content control unit 132 may be manufactured in the form of at least one hardware chip and mounted on the electronic device 100. For example, at least one of the data learning unit 131 and the content control unit 132 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be a conventional general-purpose processor application processors) or graphics-only processors (e.g., GPUs) and may be mounted on the various electronic devices 100 described above.

In this case, the data learning unit 131 and the content control unit 132 may be mounted on one electronic device 100, or may be mounted on separate electronic devices, respectively. For example, one of the data learning unit 131 and the content control unit 132 may be included in an electronic device, and the other one may be included in an external portable terminal (e.g., a smart phone) or a server. The data learning unit 131 and the content control unit 132 may provide the model information constructed by the data learning unit 131 to the content control unit 132 via wired or wireless communication.

At least one of the data learning unit 131 and the content control unit 132 may be implemented as a software module. When at least one of the data learning unit 131 and the content control unit 132 is implemented as a software module (or a program module including an instruction), the software module may be a computer-readable readable non- And may be stored in non-transitory computer readable media. Also, in this case, the at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an Operating System (OS), and some of the software modules may be provided by a predetermined application.

Hereinafter, the characteristics of the data learning unit 131 and the content control unit 132 will be described in more detail with reference to FIG. 4A to FIG.

4A, the data learning unit 131 according to some embodiments includes a data acquisition unit 131-1, a preprocessing unit 131-2, a learning data selection unit 131-3, a model A learning unit 131-4 and a model evaluation unit 131-5.

However, it is not limited thereto. According to various embodiments, the data learning unit 131 may include some of the above-described components. For example, the data learning unit 131 may include only the data acquisition unit 131-1 and the model learning unit 131-4. In addition, according to various embodiments, the data learning unit 131 may further include other components in addition to the above-described components.

The data acquiring unit 131-1 can acquire learning data necessary for analyzing a content feature according to a reaction characteristic of a user's eye. At this time, the data acquisition unit 131-1 can acquire image data, content data, and the like including user eyes photographed by the photographing unit 120 as learning data. At this time, the content data may include audio data and metadata as well as video data included in the content data.

The preprocessing unit 131-2 can pre-process the acquired learning data so that the acquired learning data can be used for analyzing the content characteristics according to the reaction characteristics of the user's eyes. The preprocessing unit 131-2 prepares the acquired data in a predetermined format so that the model learning unit 131-4, which will be described later, can use the learning data obtained for the learning for the content feature analysis according to the reaction characteristics of the user's eyes .

The learning data selection unit 131-3 can select learning data necessary for learning from the preprocessed learning data. The selected learning data may be provided to the model learning unit 131-4. The learning data selection unit 131-3 can select the learning data necessary for learning from the preprocessed learning data according to a predetermined criterion for analyzing the characteristic of the content according to the reaction characteristic of the user's eyes. For example, the learning data selection unit 131-3 can use only image data including the reaction characteristics of the predetermined user's eyes as learning data.

However, the learning data selection unit 131-3 can select some learning data among the preprocessed learning data, but this is only an example and all the preprocessed learning data can be selected. It goes without saying that the learning data selection unit 131-3 can select the learning data before the preprocessing step by the preprocessing unit 131-2.

The model learning unit 131-4 can construct a user model by learning the content characteristics according to the reaction characteristics of the user's eyes based on the learning data. At this time, the model learning unit 131-4 can analyze the content characteristics according to the reaction characteristics of the user's eyes, and learn the cumulative analysis results to construct a user model. At this time, the response feature and the content feature of the user's eye can be matched and stored in the user model. The method by which the model learning unit 131-4 constructs the user model using the learning data will be described in more detail with reference to FIG.

Also, the model learning unit 131-4 can learn the user model used for analyzing the content feature according to the reaction characteristics of the user's eyes, using the learning data. In this case, the user model may be a pre-built model. For example, the user model may be a pre-built model that receives basic learning data (e.g., reaction characteristics of a general user's eye, general content features, etc.).

The user model can be constructed considering the field of application of data analysis, the purpose of learning, or the computer performance of the device. The user model may be, for example, a model based on a neural network. For example, a model such as a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), or a Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as a user model, but the present invention is not limited thereto.

According to various embodiments, the model learning unit 131-4 can determine, as a user model to be learned, a user model having a great relation between the input learning data and the basic learning data, when a plurality of pre-built user models exist . In this case, the basic learning data may be pre-classified according to the type of data, and the user model may be pre-built according to the type of data. For example, the basic learning data may be pre-classified based on various criteria such as the creator of the learning data, the time at which the learning data was generated, the size of the learning data, the genre of the learning data,

Also, the model learning unit 131-4 can learn a user model using a learning algorithm including, for example, an error back-propagation method or a gradient descent method.

Also, the model learning unit 131-4 can learn the user model through supervised learning using, for example, learning data as an input value. Also, the model learning unit 131-4 learns, for example, the types of data necessary for analyzing the content characteristics according to the reaction characteristics of the user's eyes without much guidance, thereby obtaining the criteria for analyzing the user's driving history Through unsupervised learning, the user model can be learned. Further, the model learning unit 131-4 can learn the user model through reinforcement learning using, for example, feedback on whether the result of the user's driving history analysis based on learning is correct.

Further, when the user model is learned, the model learning unit 131-4 can store the learned user model. In this case, the model learning unit 131-4 can store the learned user model in the memory 140 of the electronic device 100 including the content control unit 132. [ Alternatively, the model learning unit 131-4 may store the learned user model in a memory of an external portable terminal or server connected to the electronic device 100 via a wired or wireless network.

In this case, the memory 140 in which the learned user model is stored may store instructions or data associated with, for example, at least one other component of the electronic device. The memory 140 may also store software and / or programs. The program may include, for example, a kernel, a middleware, an application programming interface (API), and / or an application program (or "application").

The model evaluating unit 131-5 may input the evaluation data to the user model, and if the analysis result output from the evaluation data does not satisfy the predetermined criterion, the model evaluating unit 131-5 may cause the model learning unit 131-4 to learn again . In this case, the evaluation data may be predetermined data for evaluating the user model.

For example, when the number or ratio of evaluation data for which the analysis result is not correct is greater than a preset threshold value, among the analysis results of the learned user model for the evaluation data, the model evaluation unit 131-5 satisfies the predetermined criterion It can be evaluated as not doing. For example, when a predetermined criterion is defined as a ratio of 2%, when the learned user model outputs an incorrect analysis result for evaluation data exceeding 20 out of a total of 1000 evaluation data, the model evaluation section 131-5 The learned user model can be evaluated as not suitable.

On the other hand, when there are a plurality of learned user models, the model evaluation unit 131-5 evaluates whether each of the learned user models satisfies a predetermined criterion, and determines a model satisfying a predetermined criterion as an end user model . In this case, when there are a plurality of models satisfying the predetermined criterion, the model evaluating section 131-5 can determine any one or a predetermined number of models preset in descending order of the evaluation score as an end user model.

On the other hand, the data acquisition unit 131-1, the preprocessing unit 131-2, the learning data selection unit 131-3, the model learning unit 131-4, and the model evaluation unit 131-1 in the data learning unit 131, -5) may be fabricated in at least one hardware chip form and mounted on an electronic device. For example, at least one of the data acquisition unit 131-1, the preprocessing unit 131-2, the learning data selection unit 131-3, the model learning unit 131-4, and the model evaluation unit 131-5 One may be made in the form of a dedicated hardware chip for artificial intelligence (AI), or may be fabricated as part of a conventional general purpose processor (e.g., a CPU or application processor) or a graphics dedicated processor (e.g., GPU) And may be mounted on various electronic devices.

The data acquisition unit 131-1, the preprocessing unit 131-2, the learning data selection unit 131-3, the model learning unit 131-4, and the model evaluation unit 131-5 are connected to the electronic device 100, or may be mounted on separate electronic devices, respectively. For example, some of the data acquisition unit 131-1, the preprocessing unit 131-2, the learning data selection unit 131-3, the model learning unit 131-4, and the model evaluation unit 131-5 May be included in the electronic device, and the rest may be included in the server.

At least one of the data acquisition unit 131-1, the preprocessing unit 131-2, the learning data selection unit 131-3, the model learning unit 131-4, and the model evaluation unit 131-5 Software module. At least one of the data acquisition unit 131-1, the preprocessing unit 131-2, the learning data selection unit 131-3, the model learning unit 131-4, and the model evaluation unit 131-5, (Or a program module including an instruction), the software module may be stored in a computer-readable, readable non-transitory computer readable media. Also, in this case, the at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an Operating System (OS), and some of the software modules may be provided by a predetermined application.

4A is a block diagram of the content control unit 132 according to an embodiment.

4A, a content control unit 132 according to some embodiments includes a data acquisition unit 132-1, a preprocessing unit 132-2, a recognition data selection unit 132-3, A result providing unit 132-4, and a model updating unit 132-5. However, it is not limited thereto. According to various embodiments, data classifier 1400 may include some of the components described above. For example, the content control unit 132 may include only the data acquisition unit 132-1 and the recognition result providing unit 132-4. In addition, according to various embodiments, the content control unit 132 may further include other components in addition to the above-described components.

The content control unit 132 may use at least one or more user models to estimate whether a feature of the content is contained in the content according to a criterion for determining whether the user model corresponds to the content feature according to the reaction characteristic of the user's eyes have.

The data acquisition unit 132-1 may acquire input data necessary for estimating a content feature according to a reaction characteristic of the user's eyes. In this case, the data acquiring unit can acquire the moving image data. The moving picture data may be stored in the memory 140 of the electronic device 100 or may be stored in other electronic devices such as a smart phone capable of establishing a communication relationship with the electronic device 100, , Tablet PC, server, etc.).

The preprocessing unit 132-2 can preprocess the acquired data so that the input data acquired to estimate the content characteristic can be used. The preprocessing unit 132-2 can process the acquired data so that the recognition result providing unit 132-4, which will be described later, can utilize the data acquired when the content feature is estimated from the image, into a predetermined format.

For example, the preprocessing unit 132-2 may remove noise or process the moving picture data obtained by the data obtaining unit 132-1 into a predetermined form so as to select meaningful data.

The recognition data selection unit 132-3 can select data necessary for learning from among the preprocessed input data. The selected data may be provided to the recognition result provider 132-4. The recognition data selection unit 132-3 can select some or all of the preprocessed data in order to estimate whether or not the image includes a content feature.

The recognition result provider 132-4 may apply the selected input data to the user model to estimate whether the content feature is included in the image. The recognition result providing unit 132-4 can apply the selected data to the user model by using the data selected by the recognition data selecting unit 132-3 as an input value. The recognition result providing unit 132-4 can estimate whether the image includes a content feature based on, for example, a user model corresponding to a predetermined condition among at least one or more user models. A method by which the recognition result providing unit 132-4 uses the input data to estimate the content characteristic will be described in more detail with reference to FIG.

The model updating unit 132-5 can update the user model based on the evaluation of the recognition result provided by the recognition result providing unit 132-4. For example, the model updating unit 132-5 may provide the model learning unit 131-4 with a result estimated by the content feature included in the image provided by the recognition result providing unit 132-4, The learning unit 131-4 can update the user model.

The data acquisition unit 132-1, the preprocessing unit 132-2, the recognition data selection unit 132-3, the recognition result provision unit 132-4, and the model update unit 132-4 in the content control unit 132, -5) may be fabricated in at least one hardware chip form and mounted on an electronic device. For example, the data acquisition unit 132-1, the preprocessing unit 132-2, the recognition data selection unit 132-3, the recognition result provision unit 132-4, and the model update unit 132-5 At least one may be made in the form of a dedicated hardware chip for artificial intelligence (AI) or as part of a conventional general purpose processor (e.g. CPU or application processor) or a graphics dedicated processor (e.g. GPU) It may be mounted on one electronic device.

The data acquisition unit 132-1, the preprocessing unit 132-2, the recognition data selection unit 132-3, the recognition result provision unit 132-4, and the model update unit 132-5 May be mounted on an electronic device, or may be mounted on separate electronic devices, respectively. For example, the data acquisition unit 132-1, the preprocessing unit 132-2, the recognition data selection unit 132-3, the recognition result provision unit 132-4, and the model update unit 132-5 Some may be included in the electronic device, and the rest may be included in the server.

At least one of the data acquisition unit 132-1, the preprocessing unit 132-2, the recognition data selection unit 132-3, the recognition result providing unit 132-4, and the model updating unit 132-5 May be implemented as a software module. At least one of the data acquisition unit 132-1, the preprocessing unit 132-2, the recognition data selection unit 132-3, the recognition result providing unit 132-4, and the model updating unit 132-5, When implemented in a module (or program module including an instruction), the software module may be stored in a computer-readable, non-transitory computer readable medium. Also, in this case, the at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an Operating System (OS), and some of the software modules may be provided by a predetermined application.

4B is a diagram for explaining an example in which the electronic device and the server according to an embodiment learn and recognize data by interlocking with each other.

Referring to FIG. 4B, the server 2000 may construct a disease prediction model for estimating a user model for estimating a content feature included in an image and an abnormal state (e.g., a disease state) of the eye. The electronic device 100 can estimate a content feature included in the content using the user model built in the server 2000 and estimate an abnormal state of the user's eye using the disease prediction model.

In this case, the data learning unit 2100 of the server 2000 can perform the function of the data learning unit 131 shown in FIG. 4A. That is, the data acquisition unit 2100-1, the preprocessing unit 2100-2, the learning data selection unit 2100-3, and the model learning unit 2100-4 included in the data learning unit 2100 of the server 2000 And the model evaluation unit 2100-5 shown in FIG. 4A have a data learning unit 131-1, a preprocessing unit 131-2, a learning data selection unit 131-3, a model learning unit 131-4, , And the model evaluation unit 131-5.

The data learning unit 2100 of the server 2000 may learn the user's eye reaction characteristic of the content feature to estimate the content feature included in the moving image. The data learning unit 2100 constructs a user model (e.g., the user model 300 in FIG. 3) that learns whether a user shows a specific reaction characteristic to a specific content feature and stores content characteristics according to the reaction characteristics of the user's eyes can do.

In addition, the data learning unit 2100 of the server 2000 can learn the kinds of diseases according to the states of the eyes having diseases related to the eyes. The data learning unit can construct a disease prediction model by learning the abnormal state of the eye (for example, the type of disease) according to the change of the eye state for a preset time.

The recognition result providing unit 132-4 of the electronic device 100 applies the data selected by the recognition data selecting unit 132-3 to the user model or disease prediction model generated by the server 2000, Or to estimate an abnormal condition associated with the user ' s eyes.

For example, the recognition result providing unit 132-4 transmits the input data selected by the recognition data selecting unit 132-3 to the server 2000, and the server 2000 transmits the received input data to the user model It may be applied to a disease prediction model to estimate a content feature or to estimate an abnormal condition associated with the eye. The recognition result providing unit 132-4 can receive from the server 2000 the information corresponding to the abnormal state associated with the content feature or the eye estimated by the server 2000. [

FIG. 5 is a diagram for explaining a method by which the model learning unit 131-4 constructs a user model, according to an embodiment of the present disclosure. The model learning unit 131-4 may include a response feature detection unit 530 and a content feature detection unit 540. [

First, the response feature detection unit 530 acquires image data including user's eyes with learning data. At this time, the response feature detector 530 can acquire image data including user eyes pre-processed in a predetermined format in real time, but this is only an example, and only the image data including the user's eyes showing the feature response Can be obtained.

Since the size and shape of the eye are different for each user, the reaction feature detecting unit 530 detects a user eye database 510 for storing information on the user's eyes wearing the electronic device 100, Can be used to detect changes in shape and size of the user's eyes. For example, the response feature detector 530 may detect eye blinking, eye blinding, pupil dilation, and the like of an eye shape change (e.g., distortion) as a reaction characteristic of the user's eye.

The content feature detecting unit 540 obtains the learning content. The content feature detecting unit 540 analyzes the content frame within a predetermined interval (for example, a period of about one second from the point of time when the reaction feature of the user's eye is detected) from the time when the reaction feature of the user's eye is detected, Can be detected. At this time, the content characteristic detecting unit 540 can detect an object, a color change, a brightness change, and the like included in a content frame with a content characteristic.

When a content feature is detected by the content feature detection unit 540, the content feature detection unit 540 may output the response feature of the user's eyes and the corresponding content feature to the user model 300. If the content feature detecting unit 540 does not detect the content feature, the reaction feature detecting unit 530 determines that the reaction is not caused by the content, and can remove the detected reaction feature.

The reaction characteristic detecting unit 530 and the content characteristic detecting unit 540 may repeat the above-described method to learn the characteristic of the content according to the reaction characteristic of the user's eyes.

The user model 300 may match and store the reaction characteristics and the content characteristics of the user's eyes detected by the method as described above. For example, if the user frequently winks in the scene of pinakal, the user model 300 may store that the reaction characteristic of the user's eye is "blinking" and the corresponding content characteristic is "blood, . As another example, if the user frowzed in a scene transition with a luminance change of more than 100 (e.g., increased to 13-> 113), the user model 300 would see that the reaction feature of the user's eye is " It can be stored that the corresponding content feature is "increase in brightness value 100 ". As another example, when the user totally blinks the user's eye on the red screen, the user model 300 can store the reaction characteristic of the user's eyes as "blinking of the eyes" and the corresponding degree of the content feature as "red screen". As described above, the user model 300 may store the following table.

Response characteristics of user's eyes Content Features Eyes closed Blood, knife Blinking eyes Brightness value increased by 100 Changes in the shape of the eye (distortion) Red screen appearance

The user model 300 can update the stored contents according to the reaction characteristics and the content characteristics detected by the reaction characteristic detection unit 530 and the content characteristic detection unit 540. [ For example, the user model 300 may delete existing contents or add new contents based on the results detected by the response feature detection unit 530 and the content feature detection unit 540. That is, the electronic device 100 continuously learns the content characteristics according to the reaction characteristics of the user's eyes, and builds an optimized user model for the user.

In addition, the user model 300 not only matches and stores the reaction characteristics of the user's eyes and the corresponding content features, but also stores the user's feeling information (for example, negative, positive, fearful, And the like can be stored together.

In addition, the user model correction unit 550 can modify the user model 300 according to user input. Specifically, the user model modifier 550 generates a UI for adding / deleting the reaction characteristics and the content characteristics of the user's eyes, and can modify the reaction characteristics and the content characteristics of the user's eyes according to the user's input through the UI have. For example, as shown in FIG. 7A, the user model refinement unit 550 may display a UI for adding / deleting the reaction characteristics of the user's eyes. As shown in FIG. 7B, A UI for designating a fear object of the user can be displayed.

Meanwhile, in the above-described embodiment, it has been described that the visual content characteristic and the reaction characteristic of the user's eye are matched and stored. However, this is merely an example, and the auditory content characteristic (for example, profanity, bomb sound, Sound, etc.) and the reaction characteristics of the user's eyes can be matched and stored.

FIG. 6 is a diagram for explaining a method of processing a content by the recognition result provider 132-4 of the content control unit 132 using the user model 300 according to an embodiment of the present disclosure. According to one embodiment, the recognition result providing unit 132-4 may include a user recognition unit 610, a content feature recognition unit 620, and a content reproduction control unit 630.

First, the user recognition unit 610 recognizes the user wearing the electronic device 100. [ At this time, the user recognizing unit 610 can recognize the user by photographing the iris of the user using the photographing unit 120. However, the user recognizing unit 610 can recognize the user by using another method (for example, fingerprint, Recognition, etc.).

If the user is recognized, the content feature recognition unit 620 can analyze whether the content feature stored in the user model 300 exists in the content to be reproduced. Specifically, the content feature recognition unit 620 may analyze the content frame stored in the user model 300 in advance by analyzing the content frame of the previous section from the content frame currently reproduced during the reproduction of the content. That is, the content feature recognition unit 620 can determine whether a specific object is detected, a specific color change is detected, or a specific brightness change is detected in a content frame of a previous section currently being reproduced while the content is being reproduced.

When a content feature stored in the user model 300 is detected by the content feature recognition unit 620, the content playback control unit 630 processes the content based on the control model 640. At this time, the control model 640 matches and stores the content processing methods according to the reaction characteristics and the content characteristics of the user's eye stored in the user model 300. [

In an embodiment of the present invention, when the detected content feature is a specific object and the response characteristic to a specific object is a negative response (for example, eyes closed), the content reproduction control unit 630 uses the control model 640 Thereby performing filtering processing or performing smoothing processing on the objects included in the content. For example, as shown in FIG. 8A, when the detected content feature is a knife and the response characteristic to the knife is eye-blinking, the content playback control unit 630 controls the content It is possible to perform a filtering process on the area.

In another embodiment, when the detected content feature is a specific brightness change and the user response to a specific brightness change is negative (e.g., eye fringing), the content playback control unit 630 may adjust the amount of change in a specific brightness change . For example, when the detected content characteristic is a brightness value of 100 and the response characteristic for increasing the brightness value 100 is eye distortion, the content reproduction control unit 630 decreases the brightness increase value from 100 to 70 to increase the brightness . At this time, the content reproduction control unit 630 may perform image processing to reduce the brightness increase value. However, the content reproduction control unit 630 may reduce the brightness increase value by controlling the backlight of the display 110 only by way of example.

In another embodiment, if the detected content characteristic is a specific color change and the user response to a specific color change is negative (e.g., blinking), the content reproduction control unit 630 may adjust the saturation value of the specific color change . For example, when the detected content characteristic is a red screen and the response characteristic to the appearance of a red screen is blinking, the content playback control unit 630 may adjust the saturation value so that the red screen appears as an orange screen.

Meanwhile, in the above-described embodiment, the detected content feature is a visual content feature. However, the present invention is not limited to this embodiment. In addition to the visual content feature, the auditory content feature may also be included in the technical idea of the present invention. For example, if the user's model 300 stores "snowfall" as a reaction feature corresponding to "swearing" as a content feature, when the swearing is detected in the content to be played back, the content playback control unit 630 mutes the swearing And reproduce the content.

On the other hand, in the above-described embodiment, the detected reaction characteristic of the user's eye is negative, but this is only an embodiment. If the reaction characteristic of the user's eye is positive, the reaction characteristic of the positive user's eye and the corresponding content The feature may be matched and stored in the user model 300. For example, if a pupil is expanded on a pupil's screen, the user model 300 may store the content feature as "puppy " and the corresponding response feature as" pupil dilation ". In this case, when the user subsequently inputs a command for generating a content list, the processor 130 acquires a keyword (for example, a dog) from the content characteristic corresponding to the positive reaction characteristic, A content list including features can be generated.

The processor 130 may also control the display 110 to continuously capture the user's eye to identify disease information (or health information) for the user's eye and to provide the identified disease or health information.

For example, when a user watches a moving picture using the electronic device 100, the processor 130 continuously photographs the user's eyes to check an abnormal state of the eyes to provide disease information or health information have.

This will be described in detail with reference to FIGS. 9 and 10. FIG. The processor 130 includes a user recognition unit 910, a user's eye image acquisition unit 920, a status detection unit 930, a disease determination unit 940, 950).

The user recognition unit 910 recognizes the user based on the image of the photographed user's eyes. In particular, the user recognition unit 910 can recognize a user through iris recognition, but this is merely an embodiment, and a user can be recognized through other methods.

The user's eye image acquiring unit 920 acquires image data including a user's eye photographed by the photographing unit 920. [ At this time, the user's eye image acquiring unit 920 can acquire image data including the user's eye, but this is only an example, and only the information about a part of the image data (e.g., eye or pupil) Can be obtained.

The state detection unit 930 detects the state of the user's eyes included in the image data. At this time, the state detection unit 930 can detect changes in the size, shape, color, etc. of the user's eyes based on the user's eye database 960 storing the image of the existing user's eyes.

According to one embodiment, a data learning unit (e.g., data learning unit 2100 in FIG. 4B) may generate a disease prediction model 970.

The data learning unit (e.g., the data learning unit 2100 in FIG. 4B) can learn the types of diseases according to the states of the eyes having the diseases related to the eyes in order to generate the disease prediction models 970. In this case, the data learning unit (e.g., the data learning unit 2100 of FIG. 4B) for generating the disease prediction model 970 determines whether or not the eye is abnormal based on the state change of the eye for a preset time The disease prediction model 970 can be constructed.

According to one embodiment, disease prediction model 970 may be stored in server 2000, which is external to electronic device 100. Also, according to various embodiments, the electronic device 100 may receive at least a portion of the disease prediction model 970 from the server 2000 and store the same in the memory 140 of the electronic device 100 or the like.

Based on the disease prediction model 970, the disease determination unit 940 can determine whether the disease is a disease according to the state of the user's eyes sensed by the state sensing unit 930. For example, if it is detected by the state detection unit 930 that the user's eye is congested, the disease judgment unit 940 can determine that iritis has occurred.

As another example, if it is detected that an abnormality or dilation of the pupil enlargement / reduction occurs or the reaction rate is delayed, the disease judgment unit 940 may judge that Horner's syndrome or Addis syndrome has occurred. As another example, if the pupil movement due to the head movement is detected as abnormal, the disease judgment unit 940 can judge that the lesion has occurred.

In another example, the disease judgment unit 940 can detect whether or not the nystagmus has occurred according to the movement of the eyeball. As another example, it is possible to detect whether or not an optic nerve damage or glaucoma has occurred depending on whether a visual field defect is present. At this time, the disease judging unit 940 generates a visual field map by tracking pupil movement during virtual reality (VR) contents use, and can determine whether a visual field defect exists through the generated visual field map.

The disease judgment unit 940 judges the severity of the symptom together with the size of the eye (or the pupil) detected by the state detection unit 930, the shape change, the color change, can do.

If it is determined by the disease judgment unit 940 that there is a disease in the user's eyes, the guide unit 950 may provide a guidance message including disease information on the user's eyes. For example, as shown in Fig. 10, the guidance unit 950 can provide a guidance message 1010 "Please visit the hospital soon. &Quot; At this time, the informing message may include not only the prediction of the disease but also the kind of the disease, the timing of the occurrence of the disease, and the severity of the disease.

According to one embodiment, the electronic device 100 may display the announcement message, and other electronic devices (e.g., smartphones, tablet PCs) establishing a communication relationship with the electronic device 100 may send a notification message It can also be displayed.

As such, according to various embodiments of the present disclosure, a user may use the electronic device 100 to view content, such as a moving picture, to ascertain the health state of the eye. In particular, when estimating an abnormality related to the eye using the disease prediction model () of the present disclosure, the electronic device 100 can detect the eye state change due to the eye state change for a predetermined time (for example, It is possible to improve the accuracy of the test and reduce the inconvenience of the user who has to prepare a separate time for the eye test.

11 is a flowchart for explaining a control method of an electronic device according to an embodiment of the present disclosure;

First, the electronic device 100 constructs a user model by learning content characteristics according to the reaction characteristics of the user's eyes wearing the electronic device 100 (S1110). Specifically, the electronic device 100 photographs an image including the eyes of the user wearing the electronic device while the learning contents are reproduced, and when a predetermined response characteristic to the user's eyes included in the image is sensed, A content feature included in a content frame within a predetermined interval from the point in time when the feature is sensed is acquired, and a user model is constructed by learning the predetermined response feature and the content feature.

Then, the electronic device 100 reproduces the content (S1120).

Then, the electronic device 100 detects a content feature stored in the user model from the content to be reproduced (S1130). Specifically, the electronic device 100 may analyze a content frame for a previous section of a currently played content frame to determine whether a content feature stored in the user model exists.

If the content feature included in the user model is detected (S1130-Y), the electronic device 100 processes the content to be played back based on the reaction feature of the user's eyes corresponding to the detected content feature (S1140). Specifically, the electronic device 100 may perform image processing on a content frame including the detected content feature and the detected content feature based on the response feature corresponding to the detected content feature. For example, the electronic device 100 may perform a smoothing process, a filtering process, or adjust a brightness change value or a saturation value on a content frame including a content feature.

According to various embodiments of the present invention as described above, the user can view the video image more comfortably and without irritation by performing the image processing on the content feature that the user has reacted negatively.

Meanwhile, in the above-described embodiment, the electronic device 100 constructs the user model and processes the content using the established user model, but this is merely an example, and as shown in FIG. 12, The mobile terminal 1200 connected to the electronic device 100 can build a user model and process the content using the established user model.

Specifically, while the portable terminal 1200 transmits the learning contents to the electronic device 100 and the electronic device 100 reproduces the learning contents, the electronic device 100 photographs the user's eyes, (1200). At this time, if the electronic device 100 senses a predetermined reaction characteristic to the user's eyes, the electronic device 100 can transmit the photographed image data to the portable terminal 1200.

The portable terminal 1200 can construct a user model for storing the reaction characteristics of the user's eyes and the corresponding content characteristics based on the learning contents and the photographed image data.

When the portable terminal 1200 transmits the content including the content feature included in the user model to the electronic device 100, the portable terminal 1200 transmits the content feature included in the user model and the response of the corresponding user's eyes After the image processing of the content is performed based on the feature, the image processed content can be transmitted to the electronic device 100.

13 is a flowchart for explaining a situation of estimating a content characteristic when an electronic device according to an embodiment includes a first processor and a second processor.

Referring to FIG. 13, the electronic device 100 may include a first processor 130a and a second processor 130b.

The first processor 130a controls the execution of at least one application installed in the electronic device 100 and generates a graphic for the image (e.g., live view image, photographed image, moving image, etc.) Processing can be performed. The first processor 130a may be implemented as a system on chip (SoC) integrated with functions of a central processing unit (CPU), a graphic processing unit (GPU), a communication chip, and a sensor. Also, the first processor 130a may be described as an AP (application processor) in the present specification.

The second processor 130b may estimate the region of interest of the image using the data recognition model.

Meanwhile, the second processor 130b may be manufactured as a dedicated hardware chip for artificial intelligence (AI) that performs a function of region of interest using a data recognition model. According to various embodiments, a dedicated hardware chip for artificial intelligence (AI) may include a GPU in the case of a data-aware model with visual comprehension as an elemental technology.

In addition, the electronic device 100 may further include a third processor, a fourth processor, and the like that perform the same functions as the second processor 130b.

According to various embodiments of the present disclosure, the functions performed by the first processor 130a may be stored in the memory 140 so that applications that perform various functions can perform correspondingly, and the functions performed by the second processor 130b The OS of the electronic device 100 can perform the function correspondingly.

For example, the camera application can generate a live view image and determine a data recognition model corresponding to a predetermined condition. The camera application may transmit information related to the determined data recognition model and the region of interest estimation for the server located outside the OS and / or the electronic device 100.

The OS and / or the external server can estimate the region of interest using the respective data recognition models.

According to one embodiment, the first processor 130a may play the content (S1310).

For example, the first processor 130a can reproduce a moving image stored in a memory or stream the moving image data from an external server.

The first processor 130a may transmit a frame ahead of the current frame to the second processor 130b (S1320).

The second processor 130b can estimate the content feature included in the frame by applying the received frame to the user model (S1330).

If the content characteristic is estimated, the second processor 130b may transmit the estimated content characteristic to the first processor 130a.

The first processor 130a may determine a content processing method corresponding to the estimated content feature and reflect the determined content processing method on the content. For example, the first processor 130a may perform a smoothing process, a filtering process, a brightness value adjustment, or a saturation value adjustment on a frame including the content feature (S1350).

Then, the first processor can reproduce the content to which the content processing method is applied (S1360).

14 is a flowchart for explaining a situation in which an electronic device according to an embodiment estimates a content characteristic using a server.

4B, the server 2000 according to one embodiment may include a user model.

According to one embodiment, the electronic device 100 may play the content (S1410).

For example, the electronic device 100 can reproduce a moving image stored in a memory or streaming moving image data from an external server.

In step S1420, the electronic device 100 may transmit a frame ahead of the frame to be reproduced by a preset time to the server 2000 (S1420).

Then, the server 2000 can estimate the content feature included in the frame by applying the received frame to the user model (S1430).

Once the content feature is estimated, the server 2000 may send the estimated content feature to the electronic device 100. [

The electronic device 100 can determine a content processing method corresponding to the estimated content feature and reflect the determined content processing method on the content. For example, the electronic device 100 may perform a smoothing process, a filtering process, or adjust a brightness value or a saturation value of a frame including the content feature (S1450).

Then, the electronic device 100 can reproduce the content to which the content processing method is applied (S1460).

15 is a flowchart for explaining a situation in which an electronic device according to an embodiment estimates an abnormal state related to eyes using a server.

The server 2000 according to one embodiment may include a disease prediction model.

According to one embodiment, the electronic device 100 may play the content (S1510).

For example, the electronic device 100 can reproduce a moving image stored in a memory or streaming moving image data from an external server.

Then, the electronic device 100 can take a user's eye image for a predetermined time or longer (S1520). For example, the electronic device 100 can take an eye of a user who watches a moving picture for approximately one hour while the user is viewing the moving picture. Further, according to various embodiments, the electronic device 100 may repeatedly take approximately five seconds to capture the user's eyes watching the video and again approximately five seconds after one minute while the user is viewing the video have.

The electronic device 100 may transmit the photographed snow image to the server 2000 (S1530).

Then, the server 2000 can estimate the abnormal state related to the eye by applying the received eye image to the disease prediction model (S1540). For example, if the server 2000 applies an image that gradually turns red around the user's eyes to a disease prediction model, the disease prediction model can estimate that there is iritis in the user's eyes.

The server 2000 may transmit the estimated disease content to the electronic device 100 (S1550).

The electronic device 100 can display the transmitted disease content (S1560). According to various embodiments, other electronic devices (e.g., smartphones, tablet PCs, etc.) establishing communication relationships with the electronic device 100 may display the transmitted disease content.

The above-described methods may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The above hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the disclosed embodiments may be embodied in a software program including instructions stored on a computer-readable storage medium.

The computer may include an electronic device according to the disclosed embodiments, which is an apparatus capable of calling stored instructions from a storage medium and operating according to the disclosed embodiments according to the called instructions.

The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'parity temporary' means that the storage medium does not include a signal and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily on a storage medium.

Further, the control method according to the disclosed embodiments may be provided in a computer program product. A computer program product can be traded between a seller and a buyer as a product.

The computer program product may include a computer readable storage medium having stored thereon a software program and a software program. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically via a manufacturer of an electronic device or an electronic marketplace (e.g., Google Play Store, have. For electronic distribution, at least a portion of the S / W program may be stored on a storage medium or may be created temporarily. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server for temporarily storing an SW program.

The computer program product may comprise, in a system consisting of a server and an electronic device, a storage medium of a server or a storage medium of an electronic device. Alternatively, if there is a third device (e.g. a smart phone) in communication with the server or electronic device, the computer program product may comprise a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself transmitted from the server to the electronic device or to the third device, or from the third device to the electronic device.

In this case, one of the server, the electronic device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the electronic device and the third device may execute the computer program product to distribute the method according to the disclosed embodiments.

For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on a server to control an electronic device communicatively coupled to the server to perform the method according to the disclosed embodiments.

As yet another example, a third device may execute a computer program product to control an electronic device communicatively coupled to a third device to perform the method according to the disclosed embodiment. When the third device executes the computer program product, the third device can download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a preloaded manner to perform the method according to the disclosed embodiments.

Although the present disclosure has been described with reference to certain embodiments and drawings, it is to be understood that the present disclosure is not limited to the above-described embodiments, and that various modifications and changes may be made thereto by those skilled in the art. This is possible. Therefore, the scope of the present disclosure should not be construed as being limited to the embodiments described, but should be determined by the appended claims, as well as the appended claims.

110: display 120: photographing unit
130: Processor 140: Memory
150: communication unit 160: image processing unit
170: audio output unit 180: input unit
190:

Claims (20)

1. A control method of an electronic device for providing an image worn on a head of a user,
Constructing a user model by learning a content feature according to a reaction characteristic of a user's eye wearing the electronic device; And
And processing the content to be played back based on a response feature of the user's eyes corresponding to the detected content feature, when a content feature stored in the user model is detected from the content reproduced during playback of the content Way.
The method according to claim 1,
Wherein the constructing comprises:
Capturing an image including an eye of a user wearing the electronic device while the learning contents are reproduced;
Obtaining a content feature included in a content frame within a predetermined interval from a time point when the predetermined response feature is detected for a user eye included in the image; And
And constructing the user model by learning the predetermined response characteristic and the content characteristic.
3. The method of claim 2,
The content feature may include:
Wherein the control object is at least one of an object included in a content frame within a preset interval, a brightness change of a content frame, and a color change of a content frame from the time when the predetermined response characteristic is sensed.
The method of claim 3,
The user model includes:
And stores the predetermined response characteristic corresponding to the content characteristic.
5. The method of claim 4,
Wherein the processing comprises:
Analyzing the content to be reproduced while the content is being reproduced, and determining whether a content feature stored in the user model exists; And
And if the first content feature stored in the user model exists in the content to be reproduced, the first content feature and the second content feature corresponding to the first content feature are extracted based on the first content feature and the response feature corresponding to the first content feature, And performing an image process.
6. The method of claim 5,
Wherein the processing comprises:
Wherein a filtering process or a smoothing process is performed on an object included in the content when the first content characteristic is a specific object and the response characteristic to the specific object is a negative reaction.
6. The method of claim 5,
Wherein the processing comprises:
Wherein the amount of change of the specific brightness change is adjusted when the first content characteristic is a specific brightness change and the user response to the specific brightness change is negative.
6. The method of claim 5,
Wherein the processing comprises:
Wherein the saturation value of the specific color change is adjusted when the first content characteristic is a specific color change and the user reaction to the specific color change is negative.
3. The method of claim 2,
Acquiring a keyword for a second content feature if it is determined that the user reaction to the learned second content feature is a positive response; And
And providing a list including recommended contents determined based on the obtained keywords according to a user command.
The method according to claim 1,
Accumulating and storing photographed images including user's eyes;
Analyzing an image included in the stored user's eyes to determine an abnormal state of the user's eyes; And
And providing information on an abnormal state when it is determined that the user's eye is in an abnormal state.
1. An electronic device for providing images worn on a user's head,
A display for displaying content;
A photographing unit for photographing the user's eyes;
A memory for storing a user model constructed by learning content characteristics according to a reaction characteristic of a user's eye wearing the electronic device; And
And a processor for processing the content reproduced on the basis of a reaction characteristic of the user's eye corresponding to the detected content feature, when a content feature stored in the user model is detected from the content reproduced during reproduction of the content, Device. 12. The method of claim 11,
The processor comprising:
The method includes capturing an image including a user's eye wearing the electronic device through the photographing unit while the learning content is reproduced, detecting a predetermined response feature for the user's eyes included in the image, Acquires a content feature included in a content frame within a predetermined interval from a point in time at which the content feature is set, and builds the user model by learning the predetermined response feature and the content feature.
13. The method of claim 12,
The content feature may include:
An object included in a content frame within a preset interval from the time when the predetermined response characteristic is sensed, a brightness change of the content frame, and a color change of the content frame.
14. The method of claim 13,
The user model includes:
And stores the predetermined response characteristic corresponding to the content characteristic.
15. The method of claim 14,
The processor comprising:
The method comprising the steps of: analyzing contents to be reproduced during reproduction of the contents to determine whether or not a content feature stored in the user model exists; and when a first content feature stored in the user model exists in the content to be reproduced, Wherein the image processing unit performs image processing on a content frame including the first content feature based on a feature and a response feature corresponding to the first content feature.
16. The method of claim 15,
The processor comprising:
Wherein the filtering process or the smoothing process is performed on the object included in the content when the first content characteristic is a specific object and the response characteristic to the specific object is a negative reaction.
16. The method of claim 15,
The processor comprising:
Wherein the control unit adjusts the amount of change of the specific brightness change when the first content characteristic is a specific brightness change and the user reaction to the specific brightness change is negative.
16. The method of claim 15,
The processor comprising:
Wherein the saturation value of the specific color change is adjusted when the first content characteristic is a specific color change and the user reaction to the specific color change is negative.
13. The method of claim 12,
The processor comprising:
Acquires a keyword for the second content feature when the user reaction to the learned second content feature is determined to be a positive response, and acquires the keyword for the second content feature based on the acquired keyword, And controls the display to provide the list.
12. The method of claim 11,
The processor comprising:
A memory for accumulating photographed images including user eyes in the memory and analyzing images included in the stored user eyes to determine an abnormal state of the user's eye; To control the display to provide information about the display.

Images