KR20230047871A - Content output control method using eye tracking and electronic device therefor - Google Patents

Abstract

The present invention relates to technology for controlling video and audio operations using eye-tracking cameras. An electronic device, according to an embodiment of the present invention, comprises: a camera; a display; and at least one processor operatively connected to the camera and the display. The at least one processor outputs at least one content through the display, sets an operation mode of the camera to a low power mode while outputting the at least one content, obtains image data corresponding to the user's eye area through the camera with the camera set to the low power mode, identifies the eye movement based on the acquired image data, determines whether the identified eye movement is valid, and controls parameters related to the output of the content based on the eye movement in response to determining that the eye movement is valid. In addition, various embodiments identified through the specification are possible.

Description

CONTENT OUTPUT CONTROL METHOD USING EYE TRACKING AND ELECTRONIC DEVICE THEREFOR

Various embodiments disclosed in this document relate to a technique for controlling video and audio operations using an eye tracking camera.

Virtual reality (VR) refers to a technology that artificially creates a virtual environment that is similar to a real environment but does not actually exist using an electronic device such as a computer. Augmented reality (AR) refers to a technology that synthesizes virtual objects or information in a real environment to make them look like objects existing in the original environment.

When the video is skipped forward or backward while media is being played on the mobile phone terminal device, playback can be controlled through touch input in the AR/VR device linked with the mobile phone terminal device.

To control playback of media in an AR/VR device, playback can be controlled using a controller or a separate control device such as a mobile phone or a touch key within the device.

On the other hand, in order to control media playback in conventional AR/VR devices, a separate controller is required, and some devices require a separate controller such as a mobile phone.

Therefore, in some devices without a separate control device, media playback is controlled using a separate key or gesture within the device. Pressing a separate key degrades usability, and when a gesture is used, current consumption is high and accuracy may be reduced. there is.

Various embodiments disclosed in this document may provide a device and method for controlling media reproduction in an AR/VR device without using a separate device and controlling media only with eye movements.

An electronic device according to an embodiment includes a camera, a display, and at least one processor operatively connected to the camera and the display, wherein the at least one processor outputs at least one content through the display; While outputting the at least one content, setting an operation mode of the camera to a low power mode, and obtaining image data corresponding to an eye region of a user through the camera in a state in which the camera is set to the low power mode; Identifying the eye movement based on the obtained image data, determining whether the identified eye movement is valid, and in response to determining that the eye movement is valid, based on the eye movement Parameters related to the output of the content may be controlled.

An operating method of an electronic device according to an embodiment includes outputting at least one content through a display, setting an operating mode of a camera to a low power mode while outputting the at least one content, and setting the camera to the low power consumption mode. Obtaining image data corresponding to the user's eye region through the camera in a state in which the low-power mode is set, identifying the eye movement based on the acquired image data, and the identified eye movement being effective. and controlling a parameter related to the output of the content based on the eye movement in response to determining that the eye movement is valid.

According to various embodiments disclosed in this document, convenience may be provided to a user by controlling media with a blink of an eye without using a separate device.

According to various embodiments disclosed in this document, current consumption may be reduced and accuracy may be increased by controlling media with eye blinking without a separate touch input or gesture.

In addition to this, various effects identified directly or indirectly through this document may be provided.

1 is a perspective view of an electronic device according to an exemplary embodiment.
2 is a block diagram of an electronic device according to an exemplary embodiment.
3 is a flowchart of an operation of controlling content output according to eye movement by an electronic device according to an exemplary embodiment.
4 is a flowchart of an operation of changing an operation mode of a camera by an electronic device according to an exemplary embodiment.
5 is a flowchart of an operation of outputting a user interface (UI) corresponding to a gaze area by an electronic device according to an exemplary embodiment.
6 is a flowchart of an operation of controlling an output of content based on eye movements in an electronic device according to an exemplary embodiment.
7A illustrates controlling reproduction of content based on eye movements in an electronic device according to an exemplary embodiment.
7B illustrates controlling the volume of content based on eye movements in an electronic device according to an embodiment.
7C illustrates controlling the brightness of content based on eye movements in an electronic device according to an embodiment.
8 is a diagram illustrating parameters corresponding to a UI and eye movements in an electronic device according to an embodiment.
9A is a diagram illustrating increasing the size and/or brightness of content based on eye movements in the electronic device 100 according to an embodiment.
9B is a diagram illustrating reducing the size and/or brightness of content based on eye movements in the electronic device 100 according to an embodiment.
10 is a diagram illustrating controlling a virtual object based on an eye movement in an electronic device according to an exemplary embodiment.
11 is a diagram illustrating a screen for setting or releasing a first mode in an electronic device according to an embodiment.
12 is a block diagram of an electronic device in a network environment according to various embodiments.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar elements.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that this is not intended to limit the present invention to the specific embodiments, and includes various modifications, equivalents, and/or alternatives of the embodiments of the present invention.

1 is a perspective view of an electronic device 100 according to an exemplary embodiment.

Referring to FIG. 1 , a physical state of the electronic device 100 according to an embodiment may be changed through hinges 113-L and 113-R. For example, the electronic device 100 may be in a state in which the temples of the temples are folded or the temples are unfolded through the hinges 113-L and 113-R.

According to an embodiment, the electronic device 100 includes a first light output module 101-L, a second light output module 101-R, a first display 103-L, and a second display 103-R. ), the first cameras 105-L and 105-R, the second cameras 107-L and 107-R, the third camera 109, the first printed circuit board (PCB) 111-L, 2 PCB 111-R, hinges 113-L and 113-R, first optical member 115-L, second optical member 115-R, microphones 117-L, 117-R and 117 -C (center)), speakers 119-L, 119-R, first battery 121-L, second battery 121-R, first transparent member 123-L, second transparent member (123-R). In various embodiments, the electronic device 100 may include additional components in addition to the components shown in FIG. 1 or may omit at least one of the components shown in FIG. 1 .

According to an embodiment, “R” and “L” located at the end of the identification codes described in FIG. 1 may mean components located on the right and left sides of the electronic device 100 when worn, respectively. For example, components located on the left side of the electronic device 100 when worn may be driven by power output from the first battery 121-L. Components located on the right side based on when worn may be driven by power output from the second battery 121-R.

In addition, in FIG. 1, the components located on the temple (eg, the first PCB 111-L, the second PCB 111-R), the hinges 113-L and 113-R, and the speaker 119 -L, 119-R), the first battery 121-L, and the second battery 121-R) are shown to be exposed to the outside, but this is only for convenience of description. may not be exposed to the outside.

According to an embodiment, the first light output module 101 -L and the second light output module 101 -R may be referred to as the light output module 101 . The first display 103 -L and the second display 103 -R may be referred to as the display 103 (eg, the display 230 of FIG. 2 ). The first PCB 111 -L and the second PCB 111 -R may be referred to as a PCB 111 . The first optical member 115 -L and the second optical member 115 -R may be referred to as an optical member 115 . The first battery 121 -L and the second battery 121 -R may be referred to as batteries 121 -L and 121 -R and/or battery 121 . The first transparent member 123 -L and the second transparent member 123 -R may be referred to as a transparent member 123 .

According to an embodiment, the electronic device 100 may be a wearable electronic device. For example, the electronic device 100 may be a wearable electronic device in the form of glasses (eg, augmented reality glass (AR glass), smart glass, or a head mounted device). can be However, this is only exemplary, and the present disclosure is not limited thereto. The electronic device 100 in the form of eyeglasses may operate while being worn on the user's face. The transparent member 123 may be a transparent or translucent glass plate, a plastic plate, or a polymer material so that the user can see the outside even when the electronic device 100 is worn on the user's face. In one embodiment, the first transparent member 123-L may be disposed to face the user's left eye, and the second transparent member 123-R may be disposed to face the user's right eye.

According to an embodiment, the electronic device 100 acquires (photographs) an image of the real world through the third camera 109, and the location of the acquired image or an object included in the acquired image (eg, an object, or A light output module 101, an optical member 115, and a display 103 by receiving an AR object related to a building) from another electronic device (eg, a smart phone, a computer, a tablet PC, or a server) can be provided to the user.

According to an embodiment, in order to recognize a current scene or environment viewed through the optical member 115 of the electronic device 100, the first cameras 105-L and 105-R, the second cameras 107-L, 107-R), and a third camera 109 may be utilized.

According to an embodiment, the electronic device 100 receives an audio signal through the microphones 117-L, 117-R, and 117-C, and outputs the audio signal through the speakers 119-L and 119-R. can do.

According to one embodiment, the first charging module may be disposed on the first PCB 111-L. According to an embodiment, the electronic device 100 may charge the first battery 121-L through the first charging module 112-L. According to one embodiment, the second charging module may be disposed on the second PCB 111-R. According to an embodiment, the electronic device 100 may charge the second battery 121-R through the second charging module 112-R.

2 is a block diagram of an electronic device 100 according to an embodiment.

Referring to FIG. 2 , an electronic device 100 according to an embodiment may include a processor 210, a camera 220, a display 230, and a memory 240. In various embodiments, the electronic device 100 may include additional components other than those shown in FIG. 2 or may omit at least one of the components shown in FIG. 2 .

According to an embodiment, the processor 210 may execute calculations or data processing related to control and/or communication of at least one other element of the electronic device 100 using instructions stored in the memory 240 . According to one embodiment, the processor 210 may include a central processing unit (CPU), a graphic processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, and an application. It may include at least one of a processor, an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA), and may have a plurality of cores.

According to one embodiment, the processor 210 may acquire an image through the camera 220 . According to an embodiment, the processor 210 may determine whether the user's eyes are included in the image acquired through the camera 220 . For example, a user's eye may include a pupil, an iris, a sclera, an eyebrow, and an eyelid.

According to an embodiment, the processor 210 may identify the movement of the user's eyes using at least one image including the user's eyes acquired through the camera 220 . According to an embodiment, the processor 210 may identify blinking of the user's eyes based on at least one image including the user's eyes acquired through the camera 220 . For example, the processor 210 may identify the number of eye blinks and/or the eye blink cycle based on at least one image including the user's eyes.

According to an embodiment, the processor 210 may acquire at least one image including the user's eyes through the camera 220 to identify or track the location of the user's gaze. For example, the processor 210 may identify a gaze region of the user within the display 230 based on at least one image including the user's eyes.

According to an embodiment, the processor 210 may control the output of content through the display 230 based on the user's eye movement and/or the user's gaze position. For example, the processor 210 may control the volume, reproduction speed, or reproduction of content based on the user's eye movement and/or the user's gaze position. Also, for example, the processor 210 may control the brightness of a screen displayed on the display 230 based on the user's eye movement and/or the user's gaze position.

Details related to the operation of the processor 210 will be described later with reference to FIGS. 3 to 6 .

According to an embodiment, the camera 220 may acquire (or capture) images (eg, still images and moving images). For example, the image signal processor electrically connected to the camera 220 can distinguish objects (eg, people, animals, objects) included in an image (eg, a preview image or an image stored in the memory 240) and a background. can According to one embodiment, the image signal processor may be separated from the camera 220 or implemented as part of the processor 210 . According to an embodiment, the camera 220 may include an image sensor (eg, the image sensor 1230 of FIG. 12 ). According to an embodiment, the image sensor may acquire and/or process color information.

According to an embodiment, the camera 220 may acquire at least one image frame including the user's eyes. For example, the user's eye may include at least one of an iris, a pupil, a sclera, and an eyelid.

According to an embodiment, the display 230 may generate a driving signal by converting an image signal, a data signal, an on screen display (OSD) signal, a control signal, and the like processed by the processor 210 . According to an embodiment, the display 230 may be implemented as a plasma display panel (PDP), a liquid crystal display (LCD), an organic light emitting diode (OLED), a flexible display, or the like, and may also display a 3D display. It can be implemented as a display (3D display). According to an embodiment, the display 230 is configured as a touch screen and may be used as an input device in addition to an output device.

According to an embodiment, the display 230 may output at least one piece of content or media. For example, the at least one piece of content or media may include at least one of an execution screen of an application, an image, a video, text, and a virtual object.

According to one embodiment, the memory 240 may refer to one or more memory sets. According to an embodiment, the memory 240 may store data and/or commands received from or generated by other components (eg, the processor 210 or the display 230).

3 is a flowchart of an operation of controlling content output according to eye movements by an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment.

Referring to FIG. 3 , the processor 210 according to an embodiment may output at least one content through a display (eg, the display 103 of FIG. 1 or the display 230 of FIG. 2 ) in operation 310. . According to an embodiment, the processor 210 may output at least one of an execution screen of an application, an image, a video, text, and a virtual object through the display 230 .

According to an embodiment, the processor 210 may execute at least one application installed in the electronic device 100 and display an execution screen through the display 230 . For example, the electronic device 100 may execute a video playback application and display an execution screen through the display 230 .

According to an embodiment, the processor 210 may set an operation mode of a camera (eg, the camera 220 of FIG. 2 ) to a low power mode while outputting at least one content in operation 320 . According to an embodiment, the processor 210 may set the operation mode of the camera 220 that identifies (or tracks) the position of the user's gaze to a low power mode. According to an embodiment, the low power mode may include a mode in which current consumption is reduced by lowering the resolution and/or frame rate of the camera 220 .

According to an embodiment, the processor 210 may change the operation mode of the camera 220 to a low power mode when it is determined that at least one piece of content is output through the display 230 . For example, the processor 210 may change the operation mode of the camera 220 to a low power mode when it is determined that a video playback application is being executed. Also, for example, the processor 210 may change the operation mode of the camera 220 to a low power mode when it is determined that a video is reproduced through the display 230 .

According to the above-described embodiment, the electronic device 100 according to the present disclosure can reduce current consumption by changing the operation mode of the camera 220 to a low power mode when it is determined that at least one content or media is being output. .

According to an embodiment, the processor 210 may obtain image data corresponding to the eye region of the user through the camera 220 in a state in which the camera 220 is set to the low power mode in operation 330 . For example, the user's eye region may include at least one of an eyebrow, an eyelid, a pupil, an iris, and a sclera.

According to an embodiment, the processor 210 may identify the eye movement based on the image data acquired in operation 340. According to an embodiment, the processor 210 may detect the user's gaze using eye tracking technology based on image data obtained through the camera 220 . According to an embodiment, the processor 210 may detect the user's gaze (or gaze direction) from the position and/or direction of the pupil included in the image data.

According to an embodiment, the processor 210 may identify eye blinking based on image data. For example, the processor 210 may identify at least one of an eye blink count, an eye blink cycle, and an eye blink speed. Also, for example, the processor 210 may identify at least one of the number of eye blinks occurring during a specified time period, an eye blink period within a specified time period, and an eye blink speed within a specified time period.

According to an embodiment, the processor 210 may identify eye movements based on image data acquired while the camera 220 is set to the low power mode. According to an embodiment, the electronic device 100 uses fewer infrared light emitting diodes (LEDs) that radiate infrared rays to the user's eyes and detects eye movements based on image data acquired while lowering the resolution and frame rate. can be identified. For example, the electronic device 100 identifies eye movements or eye blinks while using only one infrared LED, reducing the resolution of the camera 220 to 320 x 240, and reducing the frame rate from 96 fps to 20 fps can do.

According to an embodiment, the processor 210 may determine whether the eye movement identified in operation 350 is valid. According to an embodiment, the processor 210 may determine whether the eye movement is effective based on the image data. According to an embodiment, the processor 210 may determine whether an eye blink speed is within a specified range in order to determine whether the eye movement is effective. For example, the processor 210 may determine whether an eye blink speed is within a range of 500 ms to 100 ms in order to determine whether the eye movement is effective.

According to an embodiment, the processor 210 may obtain an eye blink rate based on the number of eye blinks for a designated time period and the eye blink cycle for a designated time period.

According to an embodiment, in response to determining that the eye movement is valid in operation 360, the processor 210 may control parameters related to content output based on the eye movement. According to an embodiment, the processor 210 may determine that the eye movement is valid when it is determined that the eye blink speed is within a specified range. For example, the processor 210 may determine that the eye movement is effective when it is determined that the eye blink speed is within a range of 500 ms to 100 ms.

According to an embodiment, when it is determined that the eye movement is valid, the processor 210 outputs the content and outputs the content based on at least one of the left and right positions of the eyes, the number of eye blinks, the eye blink cycle, and the time the eyes are closed. You can control related parameters. For example, the processor 210 determines at least one of playback speed, volume, brightness, playback movement, and playback of content that is being played or is waiting to be played, based on at least one of the left and right positions of the eyes, the number of eye blinks, and the time the eyes are closed. You can control one. A detailed description of the relationship between eye movements and parameters will be described later with reference to FIG. 8 .

4 is a flowchart of an operation of changing an operation mode of a camera by an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment. In relation to the contents of FIG. 4 , similar or overlapping contents to those described above are simplified or omitted.

Referring to FIG. 4 , the processor 210 according to an embodiment may output at least one piece of content in operation 410 . According to an embodiment, the processor 210 may output at least one of an execution screen of an application, an image, a video, text, and a virtual object through the display 230 .

According to an embodiment, the processor 210 may execute at least one application installed in the electronic device 100 and display an execution screen through the display 230 . For example, the electronic device 100 may execute a video playback application and display an execution screen through the display 230 .

According to an embodiment, the processor 210 may execute a music playback application and output sound through audio (not shown). According to an embodiment, the processor 210 may execute a music playback application and display an execution screen through the display 230 .

According to an embodiment, the processor 210 may determine whether to allow (or set) the first mode in operation 420. According to an embodiment, the processor 210 may determine whether the first mode for controlling parameters related to content output is allowed based on eye movements.

For example, the first mode may be set based on a user input to the first mode setting UI. Specifically, the processor 210 may set the first mode in response to a user's touch input or drag input to the UI allowing the first mode.

Also, for example, the processor 210 may set the first mode when a predetermined gesture is recognized. For another example, the processor 210 may set the first mode when it is determined through the camera 220 that the user gazes at a specific UI (eg, an eye icon) for a predetermined period of time or more.

According to an embodiment, the processor 210 may provide notification of whether the first mode is permitted through audio and/or UI.

According to an embodiment, the processor 210 may set an operation mode of a camera (eg, the camera 220 of FIG. 2 ) to a low power mode in operation 430 . According to an embodiment, the processor 210 may set the operation mode of the camera 220 to a low power mode in response to determining that the first mode is set. According to an embodiment, when it is determined that the first mode is set, the processor 210 may change the operation mode of the camera 220 to a low power mode that lowers the resolution and/or frame rate of the camera 220 .

5 is a flowchart of an operation of outputting a user interface (UI) corresponding to a gaze area by an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment. In relation to FIG. 5 , overlapping or similar contents to those described above are simplified or omitted.

Referring to FIG. 5 , the processor 210 according to an embodiment may acquire image data corresponding to the eye area in operation 510 . According to an embodiment, the processor 210 may obtain image data corresponding to the eye region from the camera 220 in a state in which the camera 220 is set to a low power mode.

According to an embodiment, the processor 210 may acquire an image frame including the user's eyes through the camera 220 . According to an embodiment, the processor 210 may obtain image data corresponding to the eye region from an image frame including the eye. For example, the user's eye region may include at least one of an eyebrow, an eyelid, a pupil, an iris, and a sclera.

According to an embodiment, the processor 210 may identify a gaze region within a display (eg, the display 103 of FIG. 1 or the display 230 of FIG. 2 ) in operation 520 . According to an embodiment, the processor 210 may detect the user's gaze based on image data acquired through the camera 220 . According to an embodiment, the processor 210 may detect the user's gaze from image data using eye tracking technology.

According to an embodiment, the processor 210 may detect the user's gaze (or gaze direction) from the location and/or direction of the pupil. According to an embodiment, the processor 210 may determine whether the user gazes (or gazes) at a certain area. For example, the processor 210 may determine whether the user gazes at a certain area based on a facial expression, eyelid movement, gaze fixation time, or pupil change in image data.

According to an embodiment, when it is determined that the user is gazing at a certain area, the processor 210 may determine the gazing area based on image data. For example, the processor 210 may determine the gaze region based on a facial expression, eyelid movement, gaze fixation time, or pupil change in image data.

In this document, the gaze area may refer to an area within a display on which a user gazes for a certain period of time or more.

According to an embodiment, the processor 210 may output a user interface (UI) corresponding to the gaze area in operation 530. According to an embodiment, when the gaze region is determined (or identified), the processor 210 may output a UI corresponding to the gaze region. For example, the processor 210 may display a UI related to playback control when it is determined that the gaze area is the lower area of the content execution screen. Also, for example, the processor 210 may display a UI for volume control when it is determined that the gaze area is the left area of the content execution screen. Also, for example, the processor 210 may display a UI for brightness control when it is determined that the gaze area is the right area of the content execution screen.

According to an embodiment, the processor 210 may display the UI corresponding to the gaze area so as to overlap the content. For example, when content is being reproduced, the processor 210 may overlap and display the reproduced content and the UI corresponding to the gaze area.

According to an embodiment, the processor 210 may delete the display of the UI after a second time period when it is determined that there is no control related to content output during the first time period after the UI corresponding to the gaze area is displayed.

6 is a flowchart of an operation of controlling the output of content based on eye movements in an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment.

Referring to FIG. 6 , in operation 610, the processor 210 according to an embodiment may identify eye movement based on image data corresponding to the eye region. According to an embodiment, the processor 210 may detect the user's gaze using eye tracking technology based on image data acquired through the camera 220 . According to an embodiment, the processor 210 may detect the user's gaze (or gaze direction) from the position and/or direction of the pupil included in the image data.

According to an embodiment, the processor 210 may identify eye blinking based on image data. For example, the processor 210 may identify at least one of an eye blink count, an eye blink cycle, and an eye blink speed. Also, for example, the processor 210 may identify at least one of the number of eye blinks occurring during a specified time period, an eye blink period within a specified time period, and an eye blink speed within a specified time period.

According to an embodiment, the processor 210 may obtain image data corresponding to the facial region through the camera 220 . According to an embodiment, the processor 210 may detect movement of facial muscles caused by eye blinking based on image data obtained through the camera 220 . According to an embodiment, the processor 210 may identify eye movement based on data on movement of facial muscles and image data corresponding to the eye region. According to various embodiments, a separate camera (not shown) different from the camera 220 may be used to detect facial muscle movements, and eye movements may be identified using data acquired from each camera.

According to an embodiment, the processor 210 may identify eye movements based on image data acquired while the camera 220 is set to the low power mode.

According to an embodiment, the processor 210 may determine whether the eye movement is effective in operation 620. According to an embodiment, the processor 210 may determine whether the eye movement is effective based on the image data. According to an embodiment, the processor 210 may determine whether an eye blink speed is within a specified range in order to determine whether the eye movement is effective. According to an embodiment, the processor 210 may determine that the eye movement is valid when the eye blink speed is within a specified range. According to an embodiment, the processor 210 may determine that the eye movement is invalid when it is determined that both eyes are blinking together.

According to an embodiment, the processor 210 may identify a parameter related to the output of the content based on the eye movement in operation 630 . According to an embodiment, the processor 210 may identify parameters related to brightness, volume, reproduction movement, or reproduction speed of content based on the eye movement in response to determining that the eye movement is valid. For example, the processor 210 may identify parameters related to playback movement or playback speed based on the number of eye blinks. Also, for example, the processor 210 may identify a volume or brightness parameter based on the time the eyes are closed.

According to an embodiment, the processor 210 may control output content based on parameters in operation 640 . According to an embodiment, the processor 210 may control content according to the identified parameters. For example, if a parameter related to reproduction movement is identified based on the number of eye blinks, the processor 210 may move the reproduction position of the content forward or backward according to the identified parameter. Also, for example, if a parameter related to the reproduction speed is identified based on the number of eye blinks, the processor 210 may adjust the reproduction speed of the content according to the identified parameter.

Also, for example, when the processor 210 identifies a volume-related parameter based on the time the eyes are closed, the processor 210 may increase or decrease the volume according to the identified parameter. Also, for example, if a parameter related to brightness is identified based on the time the eyes are closed, the processor 210 may increase or decrease the brightness according to the identified parameter.

FIG. 7A illustrates controlling reproduction of content based on eye movements in an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment.

Referring to FIG. 7A , the processor 210 according to an embodiment may output the first content 700 through the display 230 in a first state 710 . For example, the processor 210 may display an execution screen of a video playback application on the display 230 .

According to an embodiment, the processor 210 may determine the first gaze area 701 (eg, the lower area of the execution screen) based on image data obtained through the camera 220 . For example, the processor 210 may determine the first gaze region 701 based on a facial expression, eyelid movement, gaze fixation time, or pupil change in image data.

According to an embodiment, when it is determined that the first gaze area 701 is determined, the processor 210 may display the first UI 711 (eg, a play bar) in the second state 720. . For example, the first UI 711 may include a UI related to reproduction of content.

According to an embodiment, the processor 210 may display the first UI 711 overlapping the first content 700 in the second state 720 . According to an embodiment, the processor 210 may display the first UI 711 in an area corresponding to the first gaze area 701 in the second state 720 . For example, the processor 210 may display the first UI 711 in the lower area of the execution screen in the second state 720 .

FIG. 7B illustrates controlling the volume of content based on eye movements in an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment.

Referring to FIG. 7B , the processor 210 according to an embodiment may output the first content 700 through the display 230 in a first state 710 . For example, the processor 210 may display an execution screen of a video playback application on the display 230 .

According to an embodiment, the processor 210 may determine the second gaze area 703 (eg, the left area of the execution screen) based on image data obtained through the camera 220 . For example, the processor 210 may determine the second gaze region 703 based on facial expression, eyelid movement, gaze fixation time, or pupil change in image data.

According to an embodiment, when it is determined that the second gaze area 703 is determined, the processor 210 may display the second UI 713 (eg, a volume bar) in the second state 720. . For example, the second UI 713 may include a UI related to volume control of content.

According to an embodiment, the processor 210 may display the second UI 713 overlapping the first content 700 in the second state 720 . According to an embodiment, the processor 210 may display the second UI 713 in an area corresponding to the second gaze area 703 in the second state 720 . For example, the processor 210 may display the second UI 713 on the left area of the execution screen in the second state 720 .

7C illustrates controlling the brightness of content based on eye movements in an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment.

Referring to FIG. 7C , the processor 210 according to an embodiment may output the first content 700 through the display 230 in a first state 710 . For example, the processor 210 may display an execution screen of a video playback application on the display 230 .

According to an embodiment, the processor 210 may determine the third gaze region 705 (eg, the right region of the execution screen) based on image data acquired through the camera 220 . For example, the processor 210 may determine the third gaze region 705 based on facial expression, eyelid movement, gaze fixation time, or pupil change in image data.

According to an embodiment, if it is determined that the third gaze area 705 is determined, the processor 210 may display the third UI 715 (eg, a brightness bar) in the second state 720. . For example, the third UI 715 may include a UI related to content brightness control.

According to an embodiment, the processor 210 may overlap and display the third UI 715 on the first content 700 in the second state 720 . According to an embodiment, the processor 210 may display the third UI 715 in an area corresponding to the third gaze area 705 in the second state 720 . For example, the processor 210 may display the third UI 715 in the right area of the execution screen in the second state 720 .

8 is a diagram illustrating parameters corresponding to a UI and eye movements in an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment.

Referring to FIG. 8 , the processor 210 according to an embodiment displays a playback-related UI (eg, the first UI of FIG. 7A ) in a first area (eg, the first area 701 of FIG. 7A ) of the content execution screen. After 711) is output, parameters related to reproduction of the content may be controlled based on the movement of the user's eyes. For example, if it is determined that the left eye blinks twice, the processor 210 may move the reproduction of the content back by 10 seconds. Also, for example, if it is determined that the right eye blinks twice, the processor 210 may move the reproduction of the content forward by 10 seconds.

For example, when it is determined that the left eye blinks 3 times, the processor 210 may adjust the reproduction speed of the content to 0.5 times. Also, for example, if it is determined that the right eye blinks three times, the processor 210 may adjust the playback speed of the content to twice the speed.

Also, for example, if it is determined that both eyes are closed for one second or longer, the processor 210 may stop playing content or reproduce the stopped content.

The processor 210 according to an embodiment outputs a UI related to volume (eg, the second UI 713 of FIG. 7B ) to the second area (eg, the second area 703 of FIG. 7B ) of the content execution screen. After that, a parameter related to the volume of the content may be controlled based on the user's eye movement. For example, the processor 210 may reduce the volume by one level when it is determined that the left eye is closed for 1 second or more. Also, for example, the processor 210 may increase the volume by one level when it is determined that the right eye is closed for 1 second or longer.

The processor 210 according to an embodiment outputs a UI related to brightness (eg, the third UI 715 in FIG. 7C) to the third area (eg, the third area 705 in FIG. 7C) of the content execution screen. After that, parameters related to the brightness of the content can be controlled based on the user's eye movements. For example, if it is determined that the left eye is closed for 1 second or longer, the processor 210 may reduce the brightness by one level. Also, for example, the processor 210 may increase the brightness by one level when it is determined that the right eye is closed for 1 second or longer.

8 shows an example of the relationship between eye movements and parameters for convenience of description. However, parameters controlled according to eye movements are not limited thereto, and the output of content may be controlled based on eye movements according to various embodiments.

9A is a diagram illustrating increasing the size and/or brightness of content based on eye movements in the electronic device 100 according to an embodiment.

Referring to FIG. 9A , the electronic device 100 according to an embodiment may output first content 901 through the display 230 in a first state 910 . According to one embodiment, the first content 901 may include an execution screen of an application. For example, the first content 901 may include an execution screen of an application including at least one of alarm, weather, navigation, messenger, and memo.

According to an embodiment, the electronic device 100 may control parameters related to the output of the first content 901 based on the user's eye movements. According to an embodiment, the electronic device 100 may change the size of an image or text included in the first content 901 based on the user's eye movement.

For example, the electronic device 100 may enlarge an image or text included in the first content 901 in the second state 920 when it is determined that the right eye blinks a certain number of times or is closed for a certain period of time or more. .

According to an embodiment, the electronic device 100 may change the brightness at which the first content 901 is output based on the user's eye movement. For example, the processor 210 may increase the output brightness of the first content 901 in the second state 920 when it is determined that the right eye blinks a predetermined number of times or is closed for a predetermined period of time or more.

However, FIG. 9A illustrates an embodiment of controlling parameters according to eye movements for convenience of description, but is not limited thereto.

9B is a diagram illustrating reducing the size and/or brightness of content based on eye movements in the electronic device 100 according to an embodiment.

Referring to FIG. 9B , the electronic device 100 according to an embodiment may output second content 903 through the display 230 in a first state 910 . According to an embodiment, the second content 903 may include an execution screen of an application. For example, the second content 903 may include an execution screen of an application including at least one of alarm, weather, navigation, messenger, and memo.

According to an embodiment, the electronic device 100 may control parameters related to the output of the second content 903 based on the user's eye movements. According to an embodiment, the electronic device 100 may change the size of an image or text included in the second content 903 based on the user's eye movement.

For example, the electronic device 100 may reduce an image or text included in the second content 903 in the second state 920 when it is determined that the left eye blinks a certain number of times or is closed for a certain amount of time or more. .

According to an embodiment, the electronic device 100 may change the brightness at which the second content 903 is output based on the user's eye movement. For example, the processor 210 may reduce the output brightness of the second content 903 in the second state 920 when it is determined that the left eye blinks a predetermined number of times or is closed for a predetermined period of time.

However, for convenience of description, FIG. 9B illustrates an embodiment of controlling parameters according to eye movements, but is not limited thereto.

10 is a diagram illustrating controlling a virtual object based on an eye movement in the electronic device 100 according to an exemplary embodiment.

Referring to FIG. 10 , the electronic device 100 according to an embodiment may display a virtual object 1000 through a display 230 in a first state 1010 . According to an embodiment, the electronic device 100 may control the display 230 so that the real object 1005 overlaps the virtual object 1000 and is displayed. According to an embodiment, when the virtual object 1000 and the real object 1005 are overlapped and displayed, the processor 210 displays the real object 1005 on the display 230 as transparent or translucent so that the user can see it. can do. The virtual object 1000 may be, for example, an icon or text, or may be an image of an object (eg, a person, a building, or a natural object) existing in reality.

According to an embodiment, the electronic device 100 may control parameters related to the output of the virtual object 1000 based on the user's eye movements. According to an embodiment, the electronic device 100 may adjust the position and/or size of the virtual object 1000 in the second state 1020 based on the user's eye movement. For example, the electronic device 100 may change the position of the virtual object 1000 based on the position of the eyes and the number of blinks of the eyes. Also, for example, the electronic device 100 may change the size of the virtual object 1000 based on the position of the eyes and the number of times the eyes are closed. However, FIG. 10 illustrates an embodiment of controlling parameters according to eye movements for convenience of description, but is not limited thereto.

11 is a diagram illustrating a first screen 1100 allowing or blocking a first mode in an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment.

Referring to FIG. 11 , the electronic device 100 according to an embodiment may allow or block a first mode allowing parameters related to content output based on eye movements on the first screen 1100 to be controlled. .

According to an embodiment, the electronic device 100 may allow or block the first mode in response to a user input to the first mode setting UI 1110 included in the first screen 1100 . For example, the user input may include a touch input or a drag input to the first mode setting UI 1110 .

According to an embodiment, the first screen 1100 may include a UI for setting parameters for controlling the output of content in relation to eye movements. According to an embodiment, the processor 210 may determine a parameter for controlling the output of content based on a user input to the UI for setting parameters included in the first screen 1100 .

12 is a block diagram of an electronic device 1201 (eg, the electronic device 100 of FIGS. 1 and 2 ) within the network environment 1200 according to an embodiment.

Referring to FIG. 12 , in a network environment 1200, an electronic device 1201 communicates with an electronic device 1202 through a first network 1298 (eg, a short-range wireless communication network) or through a second network 1299. It may communicate with at least one of the electronic device 1204 or the server 1208 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 1201 may communicate with the electronic device 1204 through the server 1208 . According to an embodiment, the electronic device 1201 includes a processor 1210 (eg, the processor 210 of FIG. 2 ), a memory 1230, an input module 1250, a sound output module 1255, and a display module 1260. ), audio module 1270, sensor module 1276, interface 1277, connection terminal 1278, haptic module 1279, camera module 1280, power management module 1288, battery 1289, communication module 1290 , subscriber identification module 1296 , or antenna module 1297 . In some embodiments, in the electronic device 1201, at least one of these components (eg, the connection terminal 1278) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 1276, camera module 1280, or antenna module 1297) are integrated into a single component (eg, display module 1260). It can be.

The processor 1220, for example, executes software (eg, the program 1240) to cause at least one other component (eg, hardware or software component) of the electronic device 1201 connected to the processor 1220. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 1220 transfers instructions or data received from other components (e.g., sensor module 1276 or communication module 1290) to volatile memory 1232. , processing commands or data stored in the volatile memory 1232 , and storing resultant data in the non-volatile memory 1234 . According to one embodiment, the processor 1220 may include a main processor 1221 (eg, a central processing unit or an application processor) or an auxiliary processor 1223 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 1201 includes a main processor 1221 and an auxiliary processor 1223, the auxiliary processor 1223 may use less power than the main processor 1221 or be set to be specialized for a designated function. can The auxiliary processor 1223 may be implemented separately from or as part of the main processor 1221 .

The secondary processor 1223 may, for example, take the place of the main processor 1221 while the main processor 1221 is inactive (eg, sleep), or when the main processor 1221 is active (eg, running an application). ) state, together with the main processor 1221, at least one of the components of the electronic device 1201 (eg, the display module 1260, the sensor module 1276, or the communication module 1290) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 1223 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 1280 or communication module 1290). there is. According to an embodiment, the auxiliary processor 1223 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 1201 itself where the artificial intelligence model is executed, or may be performed through a separate server (eg, the server 1208). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, a software structure in addition to a hardware structure.

The memory 1230 may store various data used by at least one component (eg, the processor 1220 or the sensor module 1276) of the electronic device 1201 . The data may include, for example, input data or output data for software (eg, the program 1240) and commands related thereto. The memory 1230 may include a volatile memory 1232 or a non-volatile memory 1234 .

The program 1240 may be stored as software in the memory 1230 and may include, for example, an operating system 1242 , middleware 1244 , or an application 1246 .

The input module 1250 may receive a command or data to be used by a component (eg, the processor 1220) of the electronic device 1201 from an outside of the electronic device 1201 (eg, a user). The input module 1250 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 1255 may output sound signals to the outside of the electronic device 1201 . The sound output module 1255 may include, for example, a speaker or receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 1260 can visually provide information to the outside of the electronic device 1201 (eg, a user). The display module 1260 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module 1260 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 1270 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 1270 acquires sound through the input module 1250, the sound output module 1255, or an external electronic device connected directly or wirelessly to the electronic device 1201 (eg: Sound may be output through the electronic device 1202 (eg, a speaker or a headphone).

The sensor module 1276 detects an operating state (eg, power or temperature) of the electronic device 1201 or an external environment state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 1276 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 1277 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 1201 to an external electronic device (eg, the electronic device 1202). According to one embodiment, the interface 1277 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 1278 may include a connector through which the electronic device 1201 may be physically connected to an external electronic device (eg, the electronic device 1202). According to one embodiment, the connection terminal 1278 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 1279 may convert electrical signals into mechanical stimuli (eg, vibration or movement) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 1279 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 1280 may capture still images and moving images. According to one embodiment, the camera module 1280 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 1288 may manage power supplied to the electronic device 1201 . According to one embodiment, the power management module 1288 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 1289 may supply power to at least one component of the electronic device 1201 . According to one embodiment, the battery 1289 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 1290 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 1201 and an external electronic device (eg, the electronic device 1202, the electronic device 1204, or the server 1208). Establishment and communication through the established communication channel may be supported. The communication module 1290 may include one or more communication processors that operate independently of the processor 1220 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 1290 may be a wireless communication module 1292 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1294 (eg, a : a local area network (LAN) communication module or a power line communication module). Among these communication modules, the corresponding communication module is a first network 1298 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 1299 (eg, legacy It may communicate with the external electronic device 1204 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 1292 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 1296 within a communication network such as the first network 1298 or the second network 1299. The electronic device 1201 may be identified or authenticated.

The wireless communication module 1292 may support a 5G network after a 4G network and a next-generation communication technology, eg, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 1292 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 1292 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 1292 may support various requirements defined for the electronic device 1201, an external electronic device (eg, the electronic device 1204), or a network system (eg, the second network 1299). According to one embodiment, the wireless communication module 1292 may include a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 1297 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 1297 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 1297 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 1298 or the second network 1299 is selected from the plurality of antennas by, for example, the communication module 1290. can be chosen A signal or power may be transmitted or received between the communication module 1290 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 1297 in addition to the radiator.

According to various embodiments, the antenna module 1297 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 1201 and the external electronic device 1204 through the server 1208 connected to the second network 1299 . Each of the external electronic devices 1202 or 1004 may be the same as or different from the electronic device 1201 . According to an embodiment, all or part of operations executed in the electronic device 1201 may be executed in one or more external electronic devices among the external electronic devices 1202 , 1004 , or 1008 . For example, when the electronic device 1201 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 1201 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 1201 . The electronic device 1201 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 1201 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 1204 may include an internet of things (IoT) device. Server 1208 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 1204 or server 1208 may be included in the second network 1299 . The electronic device 1201 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 1236 or external memory 1238) readable by a machine (eg, electronic device 1201). It may be implemented as software (eg, the program 1240) including them. For example, a processor (eg, the processor 1220) of a device (eg, the electronic device 1201) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

As described above, an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment includes a camera (eg, the camera 220 of FIG. 2 ), a display (eg, the display of FIG. 2 ( 230)) and at least one processor (e.g., the processor 210 of FIG. 2) operatively connected to the camera and the display, wherein the at least one processor outputs at least one content through the display. and while outputting the at least one content, set an operation mode of the camera to a low power mode, and obtain image data corresponding to an eye region of the user through the camera in a state in which the camera is set to the low power mode. and identifying the eye movement based on the obtained image data, determining whether the identified eye movement is valid, and in response to determining that the eye movement is valid, to determine the eye movement. Based on this, it is possible to control parameters related to the output of the content.

According to an embodiment, at least one processor (eg, the processor 210 of FIG. 2 ) determines the number of eye blinks and the eye blink cycle for a specified time to determine whether the eye movement is effective. can be identified.

According to an embodiment, at least one processor (eg, the processor 210 of FIG. 2 ) determines the eye blink speed within a specified range based on the number of eye blinks and the eye blink cycle for the specified time period. In the case of the inner eye, it may be determined that the eye movement is effective.

According to an embodiment, the parameter related to the output of the content may include at least one of a reproduction speed, volume, brightness, and reproduction of the content.

According to an embodiment, at least one processor (eg, the processor 210 of FIG. 2 ) reproduces the content based on at least one of the left and right positions of the eyes, the number of blinks of the eyes, and the time the eyes are closed. At least one of the speed, the volume, the brightness, and the reproduction may be controlled.

According to an embodiment, at least one processor (eg, the processor 210 of FIG. 2 ) identifies a gaze position of the user based on the acquired image data, and the display (eg, the processor 210 of FIG. 2 ) according to the gaze position. A gaze area within the display 230 of the user may be determined, and the display may be controlled to display a UI corresponding to the determined gaze area.

According to an embodiment, at least one processor (eg, the processor 210 of FIG. 2 ), when the content is being reproduced, displays a message corresponding to the gaze area through the display (eg, the display 230 of FIG. 2 ). The UI may be displayed to overlap with the content.

According to an embodiment, at least one processor (eg, the processor 210 of FIG. 2 ) determines whether a control related to the output of the content is not identified for a first time after the UI corresponding to the gaze region is displayed. , the display of the UI may be deleted after the second time.

According to an embodiment, the camera may include two or more LEDs that radiate infrared rays to the user's eyes, and the at least one processor (eg, the processor 210 of FIG. 2 ) controls the resolution and frame rate of the camera. Alternatively, the operating mode of the camera may be set to a low power mode by reducing the number of activated LEDs among the LEDs.

According to an embodiment, at least one processor (eg, the processor 210 of FIG. 2 ) sets a first mode that allows the electronic device to control a parameter related to the output of the content based on the eye movement. It is determined whether or not the first mode is set, and in response to determining that the first mode is set, the operation mode of the camera (eg, the camera 220 of FIG. 2 ) may be set to the low power mode.

According to an embodiment, the at least one content may include at least one of an execution screen of an application, an image, a video, text, and a virtual object.

As described above, in the operating method of an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) according to an embodiment, at least one content is displayed through a display (eg, the display 230 of FIG. 2 ). An operation of outputting the at least one content, an operation of setting an operation mode of a camera (eg, the camera 220 of FIG. 2 ) to a low power mode while outputting the at least one content, in a state in which the camera is set to the low power mode, the Obtaining image data corresponding to the eye region of the user through a camera, identifying the eye movement based on the acquired image data, determining whether the identified eye movement is valid, and In response to determining that the eye movement is valid, an operation of controlling a parameter related to the output of the content based on the eye movement may be included.

According to an embodiment, an operating method of an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) includes the number of blinks of the eyes for a specified time period and An operation of identifying the eye blink cycle may be included.

According to an embodiment, a method of operating an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) may, based on the number of eye blinks and the eye blink cycle for the specified time, the eye blinking cycle. An operation of determining that the eye movement is valid when the blinking speed of is within a specified range may be included.

According to an embodiment, in the operating method of an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ), the parameter related to the output of the content is at least one of a playback speed, volume, brightness, and playback of the content. may contain one.

According to an embodiment, the operating method of the electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) is based on at least one of the left and right positions of the eyes, the number of blinks of the eyes, and the time the eyes are closed. and controlling at least one of the playback speed, the volume, the brightness, and the playback of the content.

According to an embodiment, an operating method of an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) includes an operation of identifying a gaze position of a user based on the obtained image data, and an operation of identifying a gaze position of a user based on the obtained image data. An operation of determining a gaze area within the display (eg, the display 230 of FIG. 2 ) and an operation of controlling the display to display a UI corresponding to the determined gaze area may be included.

According to an embodiment, a method of operating an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ), when the content is being reproduced, through the display (eg, the display 230 of FIG. 2 ). An operation of displaying the UI corresponding to the gaze area so as to overlap the content.

According to an embodiment, the operating method of an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) includes control related to the output of the content for a first time after the UI corresponding to the gaze region is displayed. When is not identified, an operation of deleting the display of the UI after a second time may be included.

According to an embodiment, in the operating method of an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ), the at least one content is at least one of an application execution screen, an image, a video, text, and a virtual object. can include

According to an embodiment, an operating method of an electronic device (eg, the electronic device 100 of FIGS. 1 and 2 ) allows the electronic device to control a parameter related to the output of the content based on the eye movement. An operation of determining whether a first mode to operate is set, and an operation of setting an operation mode of the camera (eg, the camera 220 of FIG. 2 ) to the low power mode in response to determining that the first mode is set. can include

Claims (20)

In electronic devices,
camera;
display; and
at least one processor in operative connection with the camera and the display;
The at least one processor is:
outputting at least one content through the display;
While outputting the at least one content, setting an operation mode of the camera to a low power mode;
Obtaining image data corresponding to an eye region of a user through the camera in a state in which the camera is set to the low power mode;
Identifying the movement of the eye based on the acquired image data;
determining whether the identified eye movement is valid;
An electronic device that controls a parameter related to output of the content based on the eye movement in response to determining that the eye movement is valid. The method of claim 1,
The electronic device of claim 1 , wherein the at least one processor identifies the number of eye blinks and the eye blink cycle for a specified time period in order to determine whether the eye movement is effective. The method of claim 2,
Wherein the at least one processor determines that the eye movement is effective when the eye blink rate is within a specified range based on the number of eye blinks and the eye blink cycle for the designated time period. The method of claim 1,
The parameter related to the output of the content includes at least one of a reproduction speed, volume, brightness, and reproduction of the content. The method of claim 4,
The at least one processor controls at least one of the reproduction speed, the volume, the brightness, and the reproduction of the content based on at least one of the left and right positions of the eyes, the number of blinks of the eyes, and the time the eyes are closed. , electronic devices. The method of claim 1,
The at least one processor is:
Identifying a gaze position of a user based on the obtained image data;
Determine a gaze area in the display according to the gaze position;
Controlling the display to display a UI corresponding to the determined gaze region. The method of claim 6,
Wherein the at least one processor displays the UI corresponding to the gaze region through the display so as to overlap the content when the content is being reproduced. The method of claim 6,
The electronic device, wherein the at least one processor deletes the display of the UI after a second time, when a control related to the output of the content is not identified for a first time after the UI corresponding to the gaze region is displayed. The method of claim 1,
The camera includes two or more LEDs for irradiating infrared rays to the user's eyes;
The electronic device of claim 1 , wherein the at least one processor sets an operation mode of the camera to a low power mode by reducing a resolution, a frame rate, or the number of activated LEDs among the LEDs of the camera. The method of claim 1,
The at least one processor is:
Determining whether a first mode allowing the electronic device to control a parameter related to output of the content based on the eye movement is set;
The electronic device that sets the operation mode of the camera to the low power mode in response to determining that the first mode is set. The method of claim 1,
The electronic device, wherein the at least one content includes at least one of an execution screen of an application, an image, a video, text, and a virtual object. In the operating method of the electronic device,
outputting at least one piece of content through a display;
setting an operation mode of a camera to a low power mode while outputting the at least one content;
obtaining image data corresponding to an eye region of a user through the camera in a state in which the camera is set to the low power mode;
identifying the movement of the eye based on the obtained image data;
determining whether the identified eye movement is valid; and
and controlling a parameter related to output of the content based on the eye movement in response to determining that the eye movement is valid. The method of claim 12,
and identifying the number of eye blinks and the eye blink cycle for a specified time period in order to determine whether the eye movement is effective. The method of claim 13,
and determining that the eye movement is effective when the eye blink rate is within a specified range based on the number of eye blinks and the eye blink cycle during the designated time period. The method of claim 12,
The parameter related to the output of the content includes at least one of reproduction speed, volume, brightness, and reproduction of the content. The method of claim 15
and controlling at least one of a reproduction speed, volume, brightness, and reproduction of the content based on at least one of the left and right positions of the eyes, the number of blinks of the eyes, and the time the eyes were closed. how it works. The method of claim 12,
identifying a gaze position of a user based on the obtained image data;
determining a gaze area in the display according to the gaze position; and
and controlling the display to display a UI corresponding to the determined gaze region. The method of claim 17
and displaying, through the display, the UI corresponding to the gaze region so as to overlap with the content when the content is being reproduced. The method of claim 17
and deleting the display of the UI after a second time if a control related to the output of the content is not identified for a first time after the UI corresponding to the gaze region is displayed. The method of claim 12,
determining whether a first mode allowing the electronic device to control a parameter related to output of the content based on the eye movement is set; and
and setting an operation mode of the camera to the low power mode in response to determining that the first mode is set.

Images