KR20200144196A - Electronic device and method for providing function using corneal image thereof - Google Patents

Abstract

According to various embodiments, an electronic device includes: a camera; a display; a memory; and a processor operatively connected to the camera, display, and memory. The memory, when the processor is executed, includes an instruction for performing: reception of image data including at least a part of the user′s body obtained from the camera; extraction of a cornea area of an eye from the image data; acquisition of surrounding information formed in the extracted corneal area; recognition of at least one object from the surrounding information; and provision of providing visual feedback corresponding to the recognized object through a display.

Description

Electronic devices and methods of providing functions of electronic devices using corneal images {ELECTRONIC DEVICE AND METHOD FOR PROVIDING FUNCTION USING CORNEAL IMAGE THEREOF}

This document is for an electronic device, and more specifically, an electronic device capable of photographing image data using a camera, extracting the area reflected by the user's eyes from the acquired image data, and providing various functions using this. It relates to a method of providing a function of a device.

With the development of mobile communication technology and processor technology, portable terminal devices (hereinafter, electronic devices) are able to implement various applications in addition to the conventional call function. For example, various applications such as Internet browsers, games, and multimedia have been developed and used on electronic devices.

The size of the electronic device is inevitably limited for portability, and it is difficult to enlarge the display for displaying the screens of applications. Accordingly, the user needs to look at the small-sized display in order to use functions of the electronic device.

The user can use the electronic device in various situations while on the move, during a meal, or during a meeting. At this time, as the user watches the electronic device, a lot of information in the surrounding area may be missed.

When the user watches the electronic device, it is necessary to provide information on the surrounding environment to the user.

Various embodiments of the present document are to provide an electronic device capable of acquiring an image formed on a user's cornea using a camera of an electronic device and providing various functions therefrom, and a method of providing functions of the electronic device.

An electronic device according to various embodiments includes a camera; display; Memory; And a processor operatively connected to the camera, a display, and a memory, wherein the memory stores image data including at least a part of a user's body obtained from the camera when the processor is executed. Receive, extract the cornea area of the eye from the image data, obtain surrounding information formed in the extracted cornea area, recognize at least one object from the surrounding information, and correspond to the recognized object Instructions for providing visual feedback to the display through the display may be included.

A method of providing a function of an electronic device according to various embodiments of the present disclosure includes: acquiring image data including at least a part of a user's body using a camera; Extracting a cornea area of the eye from the image data; Acquiring surrounding information formed on the extracted corneal region; Recognizing at least one object from the surrounding information; And providing visual feedback corresponding to the recognized object through a display.

According to various embodiments of the present document, an electronic device capable of acquiring an image formed on a user's cornea using a camera of an electronic device, and providing various functions therefrom, and a method of providing functions of the electronic device may be provided.

1 illustrates an electronic device in a network according to various embodiments.
2 illustrates an example of an image formed on the user's eyes when the user gazes at the electronic device.
3 is a block diagram of an electronic device according to various embodiments of the present disclosure.
4 is a flowchart of a method of providing functions of an electronic device according to various embodiments of the present disclosure.
5 illustrates an example of a method of extracting a user's eye area from an acquired image.
6 illustrates an example of a method of extracting a corneal region of an eye from an acquired image.
7 shows a histogram of the acquired corneal image.
8 shows an example of a method of forming a panoramic image by combining images formed on the cornea of both eyes.
9 illustrates an example of a method of obtaining an image formed on glasses when a user wears glasses on their eyes.
10A and 10B illustrate visual feedback provided by an electronic device according to various embodiments.
11 to 17 illustrate examples in which an electronic device provides various functions to a user using an image formed on a cornea.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1, in a network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (for example, a short-range wireless communication network), or a second network 199 It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, and a sensor module ( 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) Can be included. In some embodiments, at least one of these components (eg, the display device 160 or the camera module 180) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components may be implemented as one integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, a program 140) to implement at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and can perform various data processing or operations. According to an embodiment, as at least part of data processing or operation, the processor 120 may store commands or data received from other components (eg, the sensor module 176 or the communication module 190) to the volatile memory 132. The command or data stored in the volatile memory 132 may be processed, and result data may be stored in the nonvolatile memory 134. According to an embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and a secondary processor 123 (eg, a graphic processing unit, an image signal processor) that can be operated independently or together with the main processor 121 (eg, a central processing unit or an application processor). , A sensor hub processor, or a communication processor). Additionally or alternatively, the coprocessor 123 may be set to use less power than the main processor 121 or to be specialized for a designated function. The secondary processor 123 may be implemented separately from the main processor 121 or as a part thereof.

The coprocessor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, an application is executed). ) While in the state, together with the main processor 121, at least one of the components of the electronic device 101 (for example, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the functions or states related to. According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). have.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176). The data may include, for example, software (eg, the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

The program 140 may be stored as software in the memory 130, and may include, for example, an operating system 142, middleware 144, or an application 146.

The input device 150 may receive a command or data to be used for a component of the electronic device 101 (eg, the processor 120) from an outside (eg, a user) of the electronic device 101. The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output an sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 160 may include a touch circuitry set to sense a touch, or a sensor circuit (eg, a pressure sensor) set to measure the strength of a force generated by the touch. have.

The audio module 170 may convert sound into an electric signal or, conversely, convert an electric signal into sound. According to an embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device (for example, an external electronic device directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102) (for example, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101, or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used for the electronic device 101 to connect directly or wirelessly with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that a user can perceive through a tactile or motor sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture a still image and a video. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101. According to an embodiment, the power management module 388 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, electronic device 102, electronic device 104, or server 108). It is possible to support establishment and communication through the established communication channel. The communication module 190 operates independently of the processor 120 (eg, an application processor), and may include one or more communication processors that support direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg : A LAN (local area network) communication module, or a power line communication module) may be included. Among these communication modules, a corresponding communication module is a first network 198 (for example, a short-range communication network such as Bluetooth, WiFi direct or IrDA (infrared data association)) or a second network 199 (for example, a cellular network, the Internet, or It can communicate with external electronic devices through a computer network (for example, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip), or may be implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 in a communication network such as the first network 198 or the second network 199. The electronic device 101 can be checked and authenticated.

The antenna module 197 may transmit a signal or power to the outside (eg, an external electronic device) or receive from the outside. According to an embodiment, the antenna module may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is, for example, provided by the communication module 190 from the plurality of antennas. Can be chosen. The signal or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as part of the antenna module 197.

At least some of the components are connected to each other through a communication method (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI))) between peripheral devices and signals ( E.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be a device of the same or different type as the electronic device 101. According to an embodiment, all or part of the operations executed by the electronic device 101 may be executed by one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device 101 does not execute the function or service by itself. In addition or in addition, it is possible to request one or more external electronic devices to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the execution result to the electronic device 101. The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

Various embodiments of the present document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutes of the corresponding embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or a plurality of the items unless clearly indicated otherwise in a related context. In this document, “A or B”, “at least one of A and B”, “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “A Each of phrases such as "at least one of, B, or C" may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the component from other corresponding components, and the components may be referred to in other aspects (eg, importance or Order) is not limited. Some (eg, first) component is referred to as “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When mentioned, it means that any of the above components can be connected to the other components directly (eg by wire), wirelessly, or via a third component.

The term "module" used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic blocks, parts, or circuits. The module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (for example, the program 140) including them. For example, the processor (eg, the processor 120) of the device (eg, the electronic device 101) may call and execute at least one command among one or more commands stored from a storage medium. This makes it possible for the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here,'non-transient' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic wave), and this term refers to the case where data is semi-permanently stored in the storage medium. It does not distinguish between temporary storage cases.

According to an embodiment, a method according to various embodiments disclosed in the present document may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities. Computer program products are distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play StoreTM) or two user devices (e.g. It can be distributed (e.g., downloaded or uploaded) directly between, e.g. smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium that can be read by a device such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular number or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are sequentially, parallel, repeatedly, or heuristically executed, or one or more of the above operations are executed in a different order or omitted. Or one or more other actions may be added.

2 illustrates an example of an image formed on the user's eyes when the user gazes at the electronic device.

According to various embodiments, the electronic device 200 (eg, the electronic device 101 of FIG. 1) includes a display (eg, the display device 160 of FIG. 1 ), and provides various functions (eg, : Internet, games, multimedia, etc.) can be provided.

The user may look at the display to check the functions implemented on the display. In this case, since the viewing angle of the user is narrowed, other surrounding environments may not be recognized. For example, when a user is watching multimedia while walking, he or she may not be able to recognize an obstacle located in the direction of movement.

According to various embodiments of the present document, in a situation where the user is looking at the electronic device 200, surrounding information such as an obstacle or a person located in the front can be checked, and appropriate visual feedback may be provided to the user accordingly.

The cornea 295 is a transparent, blood vessel-free tissue on the front surface of the eyeball, and protects the eye 290 from the outside, and transmits light to the optic nerve by refracting through it. Since the cornea 295 reflects at least a portion of light, external information may be projected onto the cornea 295.

In addition, since information of a wider field of view than the user's gaze is projected onto the cornea 295, an image of a region that the user does not gaze may also be formed on the cornea 295.

Various embodiments of the present document capture a user's eye 290 looking at the electronic device 200 using a camera disposed in front of the electronic device 200, and surrounding information 295 formed on the cornea 295 therefrom. ) Can be obtained to provide various functions related to surrounding information.

3 is a block diagram of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 3, an electronic device 300 according to various embodiments may include a display 310, a camera 320, a sensor 330, a processor 340, and a memory 350. Some of them may be omitted or substituted. The electronic device 300 may include at least some of the configuration and/or functions of the electronic device 101 of FIG. 1.

The electronic device 300 may accommodate each component in a housing (not shown), a component that outputs light to the outside (for example, the display 310) and/or receives external light (for example, In the configuration of the camera 320, a part thereof may be exposed to the outside of the housing.

According to various embodiments, the display 310 displays an image, and may be implemented in one of a variety of ways, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display. . The display 310 may include at least some of the configuration and/or functions of the display device 160 of FIG. 1. The display 310 may be exposed to the outside through a part of the housing of the electronic device 300 (eg, a front BM area).

The display 310 is a touch screen display and may sense a touch or hovering (or proximity touch) input by a user's finger (or other body part) or a stylus pen.

According to various embodiments, the camera 320 may acquire external image data. The camera 320 may acquire image data using various types of image sensors such as a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS). The electronic device may arrange one or more cameras on the front side and/or the rear side of the housing, and hereinafter, unless otherwise described, image data including a region of the user's eye may be obtained using the front camera.

According to various embodiments, the memory 350 may include known volatile memory and non-volatile memory, and is not limited in specific implementation examples. The memory 350 may include at least some of the configuration and/or functions of the memory of FIG. 1. Also, the memory 350 may store at least a part of the program 140 of FIG. 1.

The memory 350 is operatively and/or electrically connected to the processor 340 and may store various instructions that may be executed by the processor 340. Such instructions may include control instructions such as arithmetic and logical operations, data movement, input/output, etc. that can be recognized by the processor 340.

According to various embodiments, the processor 340 is a component capable of performing an operation or data processing related to control and/or communication of each component of the electronic device 300, and the configuration of the processor 120 of FIG. / Or may include at least some of the functions. The processor 340 may be operatively and/or electrically connected to internal components of the electronic device 300 such as the display 310, the sensor 330, and the memory 350.

There will be no limit to the computational and data processing functions that the processor 340 can implement in the electronic device 300, but in this document, the surrounding information formed on the user's cornea is acquired, and visual feedback on the display 310 is used. Various embodiments for providing a will be described. Operations of the processor 340 to be described later may be performed by loading instructions stored in the memory 350.

According to various embodiments, the processor 340 may acquire image data using an image sensor of the camera 320. In this case, the image data includes a region of the human eye, and may include surrounding information (eg, surrounding information 280 in FIG. 2) formed on the cornea of the eye (eg, cornea 295 in FIG. 2 ). . The electronic device 300 may include a plurality of cameras 320 on the front and rear surfaces of the housing, and image data including an eye area may be obtained using the camera 320 disposed on the front side.

According to various embodiments, the processor 340 may extract a user's eye area from the acquired image data. There is no limitation on how the processor 340 extracts the user's eye area, but for example, the processor 340 uses various algorithms such as a Viola-Jones detection algorithm and a non-skin detection algorithm. The eye area can be extracted from the image data. A method of extracting the eye area by the processor 340 will be described in more detail with reference to FIG. 5.

According to various embodiments, the processor 340 may extract a cornea area from the extracted user's eye area. Because the cornea reflects at least part of the light, external information can be projected onto the cornea, and since information of a wider field of view than the user's viewing angle is projected onto the cornea, an image of an area that the user is not looking at is also formed on the cornea Can be.

According to various embodiments, the processor 340 may extract a pupil located at the center of the eye from the detected eye region, and detect a corneal region formed as an ellipse centered on the pupil. A method of extracting the corneal region by the processor 340 will be described in more detail with reference to FIG. 6.

According to various embodiments, the processor 340 may obtain image information formed in the corneal region. The processor 340 may remove noise from image information formed in the corneal region. For example, eyelashes may be extended to be included in the image of the corneal region, and the processor 340 may remove noise such as eyelashes through pixel analysis as described above.

The processor 340 may obtain surrounding information by removing noise from the image information. The surrounding information may include various objects positioned in the user's gaze direction. For example, when the user is looking at the electronic device 300, the corneal image may include a screen of the electronic device 300, and various objects such as a person and an obstacle located in front of the user may be included.

According to various embodiments, the processor 340 may determine a gaze direction (or location) of the user based on image data obtained from the camera 320. For example, the processor 340 may determine the user's gaze direction using various methods such as feature-based gaze estimation, model-based gaze estimation, iris tracking using the 3D eye model.

Here, the feature-based gaze estimation uses geometric consideration for hand-craft feature vectors mapped with auxiliary information such as eye shape and face direction (or head pose) to determine the gaze direction. This is how to detect.

Model-based gaze estimation is a method of detecting a gaze direction by determining a visual axis and an optical axis based on a 3D model that considers the center and radius of an eyeball.

According to various embodiments, the processor 340 may determine whether to output visual feedback, which will be described later, based on the user's gaze direction. For example, the processor 340 may output visual feedback corresponding to the corneal image when the user is looking at the display 310 of the electronic device 300.

According to various embodiments, the processor 340 may detect a direction of a surrounding light source from a corneal image. According to various embodiments, the processor 340 may correct distortion formed on a corneal image.

According to various embodiments, the processor 340 may extract at least one corneal region of the user's two eyes from image data acquired by the camera 320. More specifically, the processor 340 may extract at least one cornea region among the two eyes from the image data, and obtain surrounding information for each extracted cornea region.

The processor 340 may combine surrounding information acquired from at least one corneal region into one. A method in which the processor 340 combines the surrounding information of at least one corneal image into one will be described in more detail with reference to FIG. 8.

According to various embodiments, the processor 340 may recognize at least one object from surrounding information acquired from a corneal image. Since information of a wider field of view than the viewing angle of the user is projected onto the cornea, even when the user is looking at the electronic device 300, images of various objects located in front of the user may be formed on the cornea.

The processor 340 may recognize an object in the corneal image through various object recognition methods. A neural network learned on a related data set can be used to successfully recognize a particular object. Also, context awareness can only use little information to identify the subject of an image. These subjects can be used to identify different objects even if the images are similar.

According to various embodiments, the processor 340 may increase the resolution of the corneal image by using various super-resolution algorithms using deep learning in order to increase the recognition rate of an object.

According to various embodiments, the processor 340 may provide visual feedback corresponding to the recognized object. The recognized object may be an obstacle or a person located in front of the user, and the processor 340 may provide visual feedback corresponding to the recognized object through the display 310. Visual feedback to be output corresponding to the object recognized in the corneal image may be mapped and stored on the memory 350. According to various embodiments, the processor 340 may provide audio feedback or haptic feedback through a speaker together with visual feedback.

According to various embodiments, the visual feedback may include an image of the recognized object and/or a predetermined image. An example of visual feedback provided by the electronic device 300 will be described in detail with reference to FIGS. 10A and 10B.

According to various embodiments, when the recognized object is an object of a predetermined type (eg, an obstacle), the processor 340 may provide an alarm for recognizing danger as visual feedback. Accordingly, even when the user is looking at the display 310 of the electronic device 300, visual feedback on the display 310 may recognize that an obstacle is positioned in front.

According to various embodiments, the processor 340 obtains a moving speed of the electronic device 300 through the sensor 330 (eg, an acceleration sensor), and generates an alarm for recognizing a danger when the current moving speed is greater than or equal to a reference value. Can provide. This is to reduce excessive feedback provision, because there is less risk of collision with nearby obstacles in a stationary state.

According to various embodiments, the processor 340 may obtain location information of the electronic device 300 based on the recognized object and provide the location information to a location-based application (eg, a map application).

According to various embodiments, the electronic device 300 may provide augmented reality content through the display 310. The augmented reality content may include a reality image captured by the camera 320 and a virtual object generated by the electronic device 300. The processor 340 may display a virtual object to be provided to the augmented reality content based on the corneal image. For example, the processor 340 may obtain the surrounding brightness and the direction of the light source from the surrounding information, and determine the brightness, the bright area, and the shadow area of the virtual object based thereon.

According to various embodiments, the processor 340 may recognize at least one text from surrounding information and may provide information related to the text.

4 is a flowchart of a method of providing functions of an electronic device according to various embodiments of the present disclosure.

The illustrated method may be performed by the electronic device 101 of FIG. 1 and/or the electronic device 300 of FIG. 3.

In operation 410, the electronic device may acquire image data using a camera (eg, the camera 320 of FIG. 3 ). The electronic device may acquire image data including the user's eye area using a camera located in front of the housing.

In operation 420, the electronic device (eg, the processor 340 of FIG. 3) may extract a region of the user's eye from the acquired image data.

In operation 430, the electronic device may extract a cornea area from the user's eye area. Because the cornea reflects at least part of the light, external information can be projected onto the cornea, and since information of a wider field of view than the user's viewing angle is projected onto the cornea, an image of an area that the user is not looking at is also formed on the cornea. Can be.

In operation 440, the electronic device may remove noise from an image formed in the corneal region.

In operation 450, the electronic device may obtain surrounding information from an image formed in the corneal region. The surrounding information may be obtained by removing noise (eg, eyelashes, etc.) and an image of an electronic device from the corneal image.

In operation 460, the electronic device may recognize at least one object from surrounding information. When outputting visual feedback to a memory (eg, the memory 350 of FIG. 3 ), the electronic device may store objects mapped thereto in advance, and recognize an object included in the stored information.

In operation 470, the electronic device may provide visual feedback corresponding to the recognized object through a display (eg, the display 310 of FIG. 3 ). Here, the visual feedback may include an image of the recognized object and/or a predetermined image. In addition to visual feedback, the electronic device may provide audio feedback or haptic feedback through a speaker.

5 illustrates an example of a method of extracting a user's eye area from an acquired image.

According to various embodiments, the processor (eg, the processor 340 of FIG. 3) may extract a region of the user's eye from image data acquired from a camera (eg, the camera 320 of FIG. 3 ).

The method of extracting the user's eye area by the processor is not limited, but, for example, a Viola-Jones detection algorithm may be used.

The Viola-Jones detection algorithm is an algorithm that detects an object in image data based on a Haar-like feature and AdaBoost.

For example, the Viola-Jones detection algorithm measures the importance of each feature from the training image using a number of Haar-like features in the learning stage, and uses only the highest features among them to measure the importance of AdaBoost. ) Method to select only the most important features. After that, in the test step, the test image is applied as an input to the cascade structure of features previously extracted in the learning step, and the feature value is measured, and based on this, the task of detecting an object in a given image is performed. I can.

FIG. 5 shows various types of Haar-like features 500 used to find an eye area in the user's image.

As another method of extracting the user's eye area, a non-skin detection algorithm may be used. For example, the eye area may be a non-skin area within the face area, and the skin area may be detected, and the eye area may be detected among the extra skin area.

6 illustrates an example of a method of extracting a corneal region of an eye from an acquired image.

According to various embodiments, the processor (eg, the processor 340 of FIG. 3) may detect the corneal area of the eye by using image data of the user's eye area among image data. The processor may first detect the pupil region 691 from the image data of the eye and then detect the corneal region 695 by an ellipse fitting method.

When the image data includes only the eye and the peripheral area of the eye, the gray level of the pupil area 691 may be close to the darkest pixel in the image. The processor may set a maximum value (or a gray level of the darkest pixel in the image) among gray levels of pixels in the pupil region 691 as a threshold value. The processor may convert and process a thresholded binary image generated based on the threshold value to remove a region smaller than the pupil region 691. The processor may calculate the center of the remaining pixels excluding the removed area from the binary image, and determine the center area as the center of the pupil 691.

According to various embodiments, a method of detecting the pupil region 691 is not limited to the above description.

The processor may extract the corneal region 695 by analyzing the ellipse including the center of the pupil 691.

The corneal region 695 is detected by an ellipse detector looking for an ellipse parameter (c_u,c_v, r_max,r_min,?) that maximizes the response to an operator applied to the image I(u, v). Can be.

Equation (1) is the formula of the filter used to detect the corneal region 695.

[Equation 1]

Here, (cu, cv) may be the central point of the cornea (or pupil), rmax and rmin may be the maximum and minimum values of the radius, and φ may be the rotation angle of the ellipse.

The response of the filter may output an area of an elliptical arc.

The processor may detect the identified ellipse as the corneal region 695 based on the response of the filter.

According to various embodiments, a method of detecting the corneal region 695 is not limited to the above description.

7 shows a histogram of the acquired image.

According to various embodiments, a processor (eg, the processor 340 of FIG. 3) may acquire image data of a corneal area (eg, the corneal area 695 of FIG. 6) from the image data and perform noise removal. . For example, the corneal image formed in the corneal region may include not only surrounding information located in front of the user, but also the user's eyelash, which is not an image of the surrounding region formed on the cornea, and thus needs to be removed. In addition, the corneal region defined as an elliptical region as shown in FIG. 6 may also include a part of the eyelid.

7 is a histogram of a normalized corneal image. In the histogram, a section of 0 to A may be a pupil area, a section A to B may be an eyelash area, and a section D to E may be an eyelid or an area of light reflected from the eyelid.

The processor may remove noise from the eyelashes and eyelid regions through histogram analysis.

8 shows an example of a method of forming a panoramic image by combining images formed on the cornea of both eyes.

According to various embodiments, a processor (eg, the processor 340 of FIG. 3) may include at least one corneal area (eg, of the user's two eyes) from image data acquired from a camera (eg, the camera 320 of FIG. 3). : The corneal region 695 of FIG. 6) can be extracted. More specifically, the processor may extract at least one cornea region of the two eyes from the image data, and obtain surrounding information for each of the extracted cornea regions.

According to various embodiments, the processor may combine the surrounding information 881 and 882 obtained in each corneal region into one.

For example, when two eyes of a user are photographed in the image data, the two eyes and a camera can be viewed as a catadioptric stereo system. The processor may determine a relative orientation between the binocular cornea and the camera, and may determine the positions of the corneal images 881 and 882. The obtained left eye corneal image 881 may have an epipolar curve corresponding to the right eye corneal image 882. The processor may generate one surrounding information 880 by combining both images based on the epipolar curve of the left eye corneal image 881 and the right eye corneal image 882.

According to various embodiments, the processor may combine information of a plurality of consecutive corneal images in order to obtain surrounding information in a wide-angle view.

For example, when the user moves the gaze, different images around the cornea may be acquired in each frame. The processor may extract a corneal region for a plurality of sequentially acquired frames, acquire surrounding information, and combine the surrounding information of each frame to obtain surrounding information of a wide angle.

By using a plurality of spherical panorama images, it is possible to minimize distortion on an overlapped spherical panorama image. The pixels of each spherical panorama can be calculated from the 3D coordinates and orientation of the cornea, and parameters obtained therefrom can be used to combine each panorama image.

9 illustrates an example of a method of obtaining an image formed on glasses when a user wears glasses on their eyes.

According to various embodiments, when a user looks at the electronic device 900 while wearing an accessory 970 such as glasses, goggles, and a helmet, the camera of the electronic device 900 is worn by the user, not by the human eye. An image formed on the accessory 970 may be obtained. When the user wears the accessory 970, the corneal image cannot be obtained because the eye image is not acquired by the front camera of the electronic device, but surrounding information may also be displayed on the accessory 970.

The processor may acquire surrounding information by using the image 980 formed on the accessory and extract an object from the surrounding information.

10A and 10B are diagrams illustrating visual feedback provided by an electronic device according to various embodiments of the present disclosure.

According to various embodiments, the processor (eg, the processor 340 of FIG. 3) recognizes at least one object from surrounding information acquired in the corneal region, and provides visual feedback corresponding to the recognized object through the display 1010. Can provide.

According to various embodiments, the visual feedback may be a predetermined image 1061 for requesting the user to pay attention to the surrounding environment. Referring to FIG. 10A, an image 1061 provided as visual feedback may include image and text information requiring attention of a user.

In a situation in which the user looks at the display 1010 of the electronic device 1000 and misses an object (eg, an obstacle, a person) in the vicinity, as the visual feedback 1061 is provided, the user turns his gaze and You can make them aware of the situation.

Referring to FIG. 10B, the visual feedback may be an image 1062 of a recognized object. For example, when there is an obstacle in front while the user is looking at the display 1010 of the electronic device 1000 while moving, an image of the obstacle may be formed on the user's cornea. The processor may display the image 1062 of the acquired object (eg, an obstacle) on the display 1010, so that the user can recognize an object nearby even when the user is looking at the electronic device.

The processor may display the visual feedback including the image 1062 of the object on an area of the currently executing application screen, such as a pop-up window, or may display a semi-transparent layer on the upper part of the application screen.

11 to 17 illustrate examples in which an electronic device provides various functions to a user using an image formed on a cornea.

11 shows a situation in which a user is looking at an electronic device while moving (or walking). As described above, since the user is looking at the electronic device 1100, it may not be recognized that the obstacle 1181 located in the vicinity is located in the movement path.

According to various embodiments, the processor acquires the surrounding information 1180 through the corneal image, and the object of the surrounding information 1180 is a predetermined dangerous object, for example, a person, a street light, a crack on the road, a pit, etc. In this case, visual feedback 1160 may be provided to the user through the display. The visual feedback 1160 may include, for example, text or image information for notifying that there is an obstacle ahead. According to another embodiment, the processor may display an image of an object acquired through the corneal image on the display.

According to various embodiments of the present disclosure, the electronic device may store an image to be displayed corresponding to an object acquired from a corneal image in a memory.

According to various embodiments, the electronic device detects the movement of the user using at least one sensor (eg, the sensor 330 in FIG. 3) or the rear camera, and recognizes an obstacle when the movement speed of the user is more than a predetermined value. In this case, an alarm (or visual feedback 1160) for recognizing the danger may be provided. This is to minimize the risk caused by obstacles in the stop situation and to minimize the stress that the alarm gives while using the electronic device.

According to various embodiments, the electronic device detects a user's gaze using an image acquired through a front camera, and when it is detected that the user's gaze is looking at a display, the electronic device generates an alarm 1160 for recognizing the danger. Can provide.

According to various embodiments, when the user is watching a specific point (eg, navigation) for a long time while driving, the electronic device may detect an unexpected situation using a corneal image and provide feedback to the user. For example, when the user is watching a navigation, the processor may detect a traffic light, a vehicle, etc. using a corneal image, and provide an alarm through a display or a sound alarm through a speaker to the user.

The processor may further acquire an image of a rear view mirror using a camera, analyze a corneal image and an object in the image of the rear mirror, and recognize a traffic condition.

12 shows an embodiment of providing a location-based service using a corneal image.

The electronic device 1200 may execute an application (eg, a map application) that provides various location-based services. The location-based application may acquire the current location of the electronic device and provide various information based on the current location.

According to various embodiments, the processor may recognize an object in the surrounding information 1280 acquired from the corneal image 1290 and determine a current location of the electronic device based on the object.

For example, when text such as an address or building name indicating specific location information or a landmark building 1281 is recognized by the user's eyes, the electronic device may check the location of the corresponding object. The electronic device may provide the current location through a location-based application.

The electronic device may utilize the corneal image for indoor positioning as well.

13 illustrates an embodiment of providing information on surrounding objects using a corneal image.

Referring to FIG. 13, a user may use the electronic device 1300 while eating food. While the user is looking at the electronic device 1300, an image of the food 1381 may also be formed in the user's eye, and the electronic device may acquire the food object 1380 on the user's corneal image 1390. .

The electronic device 1300 may provide information related to the recognized food 1381 as visual feedback 1360 on the display. For example, you can check the profile of the food (e.g., nutrients, calories, etc.) and immediately display it on the display. Can provide.

14 illustrates an embodiment of recognizing surrounding brightness using a corneal image.

According to various embodiments, the corneal image may also be used to provide augmented reality.

Referring to FIG. 14, the electronic device 1400 may acquire ambient brightness and a direction of a light source based on a user's corneal image 1490. When the surrounding light source 1481 is shining on the user's eyes, an image of the light source is formed on the corneal image, and the electronic device may recognize the surrounding brightness and the direction of the light source through analysis of the corneal image.

When providing the augmented reality content 1460 based on the brightness and the direction of the light source, the electronic device may determine the brightness of the augmented reality object and determine the location of the shadow. For example, referring to FIG. 14, when a sofa is displayed as an augmented reality object in an indoor empty space, the electronic device recognizes the brightness of the room and the direction of the light source from the corneal image, You can determine the location of the shadow.

15 illustrates an embodiment of recognizing surrounding text using a corneal image.

According to various embodiments, the electronic device 1500 may recognize at least one text 1581 from the surrounding information 1580 obtained from the corneal image 1590 and provide information related to the text 1581. .

Referring to FIG. 15, a signboard of a nearby store may be formed on the corneal image 1590, and the electronic device may provide information, advertisements, etc. related to the corresponding store by recognizing the text of the corneal image 1590.

When the name of a specific product or the like is recognized on the corneal image, the electronic device may provide personalized information or a recommendation event for the product or related product.

16 illustrates an embodiment of recognizing another person's gaze using a corneal image.

Referring to FIG. 16, when a user concentrates on a game or a multimedia application using the electronic device 1600, recognition of the surrounding environment may be neglected.

The electronic device recognizes the gaze 1680 of another person through the corneal image 1690 and, when it is determined that the other person is looking at the user of the electronic device, provides an alarm 1660 to warn it on the display. I can. The electronic device 1690 may provide the alarm 1660 as a pop-up window on the running application screen.

17 illustrates an embodiment of guiding a user to execute other functions using a corneal image.

Referring to FIG. 17, the electronic device 1700 may recognize that a user is currently boarding an airplane using a corneal image 1790. For example, surrounding information including an object 1780 such as a window of an airplane may be formed in the corneal image 1790.

In this case, the electronic device 1700 may provide a visual feedback 1760 guiding a necessary function to be executed according to the current location and state of the user. For example, when the electronic device is not currently operating in the airplane mode, the electronic device may provide an alarm 1760 guiding execution of the airplane mode.

The electronic device may collect and store current state and surrounding information using the corneal image. In this case, it may be suggested that users execute the same function in the future in the same environment. For example, when it is recognized that the current location is within an airplane from the corneal image, and the user executes the airplane mode at this time, the current location and the executed function may be mapped and stored in a memory. Thereafter, when a pre-stored location or situation is recognized through the corneal image, the electronic device may provide a recommendation to the user based on a previously executed function.

An electronic device according to various embodiments includes a camera; display; Memory; And a processor operatively connected to the camera, a display, and a memory, wherein the memory stores image data including at least a part of a user's body obtained from the camera when the processor is executed. Receive, extract the cornea area of the eye from the image data, obtain surrounding information formed in the extracted cornea area, recognize at least one object from the surrounding information, and correspond to the recognized object Instructions for providing visual feedback to the display through the display may be included.

According to various embodiments, the instructions may include instructions for displaying at least one of an image of the recognized object and a predetermined image through the display.

According to various embodiments, the instructions may include instructions for providing an alarm for recognizing danger as the visual feedback when the recognized object is a predetermined type of object.

According to various embodiments, the instruction further comprises a sensor for measuring a moving speed of the electronic device, wherein the instructions provide an alarm for recognizing the danger when the moving speed measured by the sensor is higher than a predetermined value. Can include.

According to various embodiments, the instructions may include instructions for obtaining location information of the electronic device based on the recognized object, and providing the obtained location information to an executed location-based application.

According to various embodiments, the instructions may include instructions for displaying augmented reality content on the display and displaying the visual feedback on the augmented reality content.

According to various embodiments, the instructions may include instructions for obtaining ambient brightness from the surrounding information.

According to various embodiments, the instructions may include instructions for recognizing at least one text from the surrounding information and providing information related to the text.

According to various embodiments, the camera may be disposed on the front surface of the housing of the electronic device.

According to various embodiments, the instructions may include instructions for extracting at least one corneal region of the user's eye from the received image data.

According to various embodiments, the instructions may include instructions for detecting a position of a user's gaze from the image data, and providing the visual feedback when the position of the gaze faces at least a part of the electronic device.

A method of providing a function of an electronic device according to various embodiments of the present disclosure includes: acquiring image data including at least a part of a user's body using a camera; Extracting a cornea area of the eye from the image data; Acquiring surrounding information formed on the extracted corneal region; Recognizing at least one object from the surrounding information; And providing visual feedback corresponding to the recognized object through a display.

According to various embodiments, the providing of the visual feedback may include displaying at least one of an image of the recognized object and a predetermined image through the display.

According to various embodiments, when the recognized object is a predetermined type of object, the providing of the visual feedback may include providing an alarm for recognizing danger as the visual feedback.

According to various embodiments, the operation of measuring a movement speed of the electronic device using a sensor further includes, and the operation of providing the visual feedback is performed for recognizing the danger when the movement speed measured through the sensor is more than a predetermined value. It may include an operation of providing an alarm.

According to various embodiments of the present disclosure, the providing of the visual feedback may include acquiring location information of the electronic device based on the recognized object; And providing the obtained location information to an executed location-based application.

According to various embodiments, the operation of acquiring the surrounding information may include an operation of acquiring the surrounding brightness from the surrounding information.

According to various embodiments, the providing of the visual feedback includes: recognizing at least one text from the surrounding information; And providing information related to the text.

According to various embodiments, the operation of extracting the corneal region of the eye may include extracting the corneal region of the eye of at least one user from the received image data.

According to various embodiments, the operation of detecting a position of a user's gaze from the image data further includes, wherein the providing of the visual feedback includes the visual feedback when the position of the gaze faces at least a portion of the electronic device. It may include an operation to provide.

Claims (20)

In the electronic device,
camera;
display;
Memory; And
A processor operatively connected with the camera, display and memory,
The memory, when the processor is executed,
Receiving image data including at least a part of the user's body obtained from the camera,
Extracting the cornea area of the eye from the image data,
Acquire surrounding information formed on the extracted corneal region,
Recognizing at least one object from the surrounding information, and
An electronic device comprising instructions for providing visual feedback corresponding to the recognized object through the display.
The method of claim 1,
The above instructions,
An electronic device comprising instructions for displaying at least one of the recognized object image and a predetermined image through the display.
The method of claim 1,
The above instructions,
When the recognized object is a predetermined type of object, an electronic device comprising instructions to provide an alarm for recognizing danger as the visual feedback.
The method of claim 3,
Further comprising a sensor for measuring the moving speed of the electronic device,
The above instructions,
Electronic device comprising instructions to provide an alarm for recognizing the danger when the moving speed measured by the sensor is more than a predetermined value.
The method of claim 1,
The above instructions,
Based on the recognized object, obtains location information of the electronic device,
An electronic device comprising instructions for providing the obtained location information to an executed location-based application.
The method of claim 1,
The above instructions,
Display augmented reality content on the display,
An electronic device comprising instructions to display the visual feedback on the augmented reality content.
The method of claim 1,
The above instructions,
Electronic device including instructions for obtaining the brightness of the surroundings from the surrounding information.
The method of claim 1,
The above instructions,
Recognizing at least one text from the surrounding information,
An electronic device including instructions for providing information related to the text.
The method of claim 1,
Wherein the camera is disposed on the front surface of the housing of the electronic device.
The method of claim 1,
The above instructions,
An electronic device comprising instructions for extracting at least one corneal region of the user's eye from the received image data.
The method of claim 1,
The above instructions,
Detecting the position of the user's gaze from the image data,
The electronic device comprising instructions for providing the visual feedback when the gaze position is toward at least a portion of the electronic device.
In the method for providing functions of an electronic device,
Obtaining image data including at least a part of a user's body using a camera;
Extracting a cornea area of the eye from the image data;
Acquiring surrounding information formed on the extracted corneal region;
Recognizing at least one object from the surrounding information; And
And providing visual feedback corresponding to the recognized object through a display.
The method of claim 12,
The operation of providing the visual feedback,
And displaying at least one of the recognized object image and a predetermined image through the display.
The method of claim 12,
The operation of providing the visual feedback,
And when the recognized object is a predetermined type of object, providing an alarm for recognizing danger as the visual feedback.
The method of claim 14,
Further comprising an operation of measuring the moving speed of the electronic device using the sensor,
The operation of providing the visual feedback,
And providing an alarm for recognizing the danger when the moving speed measured by the sensor is more than a predetermined value.
The method of claim 12,
The operation of providing the visual feedback,
Acquiring location information of the electronic device based on the recognized object; And
And providing the obtained location information to an executed location-based application.
The method of claim 12,
The operation of obtaining the surrounding information,
And obtaining brightness of the surroundings from the surrounding information.
The method of claim 12,
The operation of providing the visual feedback,
Recognizing at least one text from the surrounding information; And
And providing information related to the text.
The method of claim 12,
The operation of extracting the corneal region of the eye,
And extracting a corneal region of at least one user's eye from the received image data.
The method of claim 12,
Further comprising the operation of detecting the gaze position of the user from the image data,
The operation of providing the visual feedback,
And providing the visual feedback when the gaze position faces at least a portion of the electronic device.

Images