WO2022131659A1 - Electronic apparatus and controlling method thereof - Google Patents

Abstract

An electronic apparatus according to an embodiment comprises: a screen; a magnetometer sensor which acquires magnetic field data in front of the screen; and a processor, wherein the processor: when an object hovering over the screen is detected on the basis of the magnetic field data, determines a distance threshold value corresponding to the object on the basis of the size and type of the object; and when the distance between the object and the screen is less than the distance threshold value, performs an operation corresponding to the position of the object.

Description

Electronic device and its control method

The present invention relates to an electronic device that can be controlled using an object without a touch, and a method for controlling the same.

BACKGROUND Electronic devices such as e-book readers, smart phones, tablet computers, and the like include touch sensor panels, which are becoming more and more popular due to the ease and versatility of operation on the touch screen. In an electronic device, an active digitizer sheet may be integrated with a touch sensor panel to sense a touch input of a stylus.

1 illustrates an electronic device 11 with a stylus 13 . The user may provide the touch input 12 to the electronic device 11 by holding the stylus 13 like a pen or pencil. The active digitizer sheet consumes power of the electronic device 11 . Accordingly, the battery of the electronic device 11 is rapidly consumed.

Also, in general, the stylus 13 may be inserted into the cavity of the electronic device 11 . As the stylus 13 is inserted into the cavity of the electronic device 11 , the weight of the electronic device 11 increases, and the manufacturing cost of the electronic device 11 also increases due to additional hardware such as the stylus 13 .

Moreover, removal/insertion of the stylus 13 in the thin and slim electronic device 11 may be difficult for the user, which degrades the user experience while using the stylus 13 . In general, thin mechanical elements are used to design the stylus 13 where there is a risk that the stylus 13 is damaged or broken. Accordingly, it would be desirable to at least provide a useful alternative.

Provided are an electronic device that can be controlled using an object such as ferromagnetic without touching the electronic device, and a control method thereof.

An electronic device according to an embodiment includes a screen; a magnetometer sensor for acquiring magnetic field data in front of the screen; and a processor, wherein when detecting an object hovering over the screen based on the magnetic field data, a distance threshold value corresponding to the object based on the size and type of the object and if the distance between the object and the screen is less than the distance threshold, an operation corresponding to the position of the object is performed.

The processor may be configured to detect that the object is hovering over the screen when determining that a rate of change of the magnetic field in front of the screen exceeds a magnetic field threshold based on the magnetic field data.

The electronic device may include a speaker; and a microphone; wherein the processor controls the speaker to output a sound signal toward the object when detecting the object, and is reflected from the object after being output from the speaker and received by the microphone The size of the object may be determined based on the data of the sound signal.

The processor may determine the type of the object based on an output of a machine learning (ML) model to which the magnetic field data is input.

The processor may determine the position of the object based on the speed of the object, the magnetic field data, and the direction of the electronic device.

The electronic device may include an accelerometer sensor; and a gyro sensor, wherein the processor determines the speed of the object based on accelerometer data output from the accelerometer sensor, and determines the direction of the electronic device based on direction data output from the gyro sensor. can

The processor is configured to determine a position vector of the object with respect to a three-dimensional Cartesian coordinate system based on the speed of the object and the magnetic field data, and the object relative to the screen based on the position vector of the object and the direction of the electronic device position can be determined.

The processor may determine a point indicated by the object on the screen based on the position of the object, determine a user interface data item including the point, and perform an operation corresponding to the user interface data item .

The processor may determine a movement form of the object on the screen based on the position of the object, and control the screen to display a character corresponding to the movement form of the object.

The processor is configured to change the user profile of the electronic device from the first user profile corresponding to the first portion to the second portion when the position of the object moves from the first portion of the screen to the second portion of the screen. may be changed to a second user profile corresponding to .

In the control method of an electronic device according to an embodiment including a screen and a magnetometer sensor for acquiring magnetic field data in front of the screen, when detecting an object hovering over the screen based on the magnetic field data, the determine a distance threshold value corresponding to the object based on the size and type of the object; and performing an operation corresponding to the position of the object when the distance between the object and the screen is less than the distance threshold.

Detecting the object may include detecting that the object hovers over the screen when it is determined based on the magnetic field data that the rate of change of the magnetic field in front of the screen exceeds a magnetic field threshold. .

The electronic device may include a speaker; and a microphone, wherein the method of controlling the electronic device includes: when detecting the object, controlling the speaker to output a sound signal toward the object; It may further include; determining the size of the object based on the data of the sound signal reflected from the object after being output from the speaker and received by the microphone.

The control method of the electronic device may further include determining the type of the object based on an output of a machine learning (ML) model to which the magnetic field data is input.

The method of controlling the electronic device may further include determining the position of the object based on the speed of the object, the magnetic field data, and the direction of the electronic device.

According to an electronic device and a method for controlling the same according to an embodiment, a stylus and an activated digitizer sheet are provided by providing control of an electronic device using an object, such as a ferromagnetic, without a touch based on an output of a pre-installed sensor. can be excluded, the size, weight, and manufacturing cost of the electronic device can be reduced, and power consumption of the electronic device can be significantly improved.

1 is a diagram illustrating an electronic device having a stylus according to the related art.

2A is a control block diagram of an electronic device according to an exemplary embodiment.

2B is a block diagram of an operation controller for performing an operation of an electronic device according to an exemplary embodiment.

3 and 4 are flowcharts illustrating a method of controlling an operation of an electronic device according to an exemplary embodiment.

5A is a diagram illustrating a signal received by a microphone of an electronic device according to an exemplary embodiment.

5B is a diagram illustrating a case in which an electronic device determines a size of an object hovered over a screen according to an exemplary embodiment.

6A and 6B are diagrams illustrating a recurrent neural network (RNN) used by an electronic device to identify a type of an object hovered over a screen, according to an exemplary embodiment.

7 is a diagram illustrating a plot of a volume of an object with respect to a height of the object from a screen of an electronic device according to an exemplary embodiment.

8 is a diagram illustrating a case in which an electronic device determines a speed of an object hovered over a screen according to an exemplary embodiment.

9 is a diagram illustrating a case in which an electronic device calculates a direction according to an exemplary embodiment.

10 is a diagram illustrating a position vector of an object in a 3D Cartesian coordinate system and a position of the object with respect to a screen in an electronic device according to an exemplary embodiment.

11 is a diagram illustrating an exemplary scenario in which an electronic device receives an alphabet based on a position of an object, according to an embodiment.

12 is a diagram illustrating an exemplary scenario in which an electronic device switches a user profile based on a location of an object, according to an embodiment.

13 is a diagram illustrating an exemplary scenario in which an electronic device instructs to wash a user's hand based on a location of an object, according to an embodiment.

The configuration shown in the embodiments and drawings described in this specification is only a preferred example of the disclosed invention, and there may be various modifications that can replace the embodiments and drawings of the present specification at the time of filing of the present application.

Throughout this specification, when a part is "connected" with another part, this includes not only a case in which it is directly connected but also a case in which it is indirectly connected, and the indirect connection refers to being connected through a wireless communication network. include

In addition, the terminology used herein is used to describe the embodiments, and is not intended to limit and/or limit the disclosed invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, element, or a combination thereof described in the specification exists, but one or more other features It does not preclude the possibility of the presence or addition of numbers, steps, operations, components, elements, or combinations thereof.

In addition, terms including an ordinal number, such as "first", "second", etc. used herein may be used to describe various elements, but the elements are not limited by the terms, and the terms are It is used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

In addition, terms such as "~ part", "~ group", "~ block", "~ member", and "~ module" may mean a unit for processing at least one function or operation. For example, the terms may refer to at least one hardware such as a field-programmable gate array (FPGA) / application specific integrated circuit (ASIC), at least one software stored in a memory, or at least one process processed by a processor. have.

The signs attached to each step are used to identify each step, and these signs do not indicate the order between the steps, and each step is performed differently from the stated order unless the context clearly indicates a specific order. can be

An embodiment of the disclosed invention provides a method for controlling an operation of an electronic device using an object without a touch. The method includes detecting, by the electronic device, an object hovered over a screen of the electronic device. The method includes obtaining, by the electronic device, data from a first set of sensors (a first set of sensors). The method includes determining, by the electronic device, a threshold value of a distance of the object to a screen of the electronic device based on data received from a first set of sensors. The method includes obtaining, by the electronic device, data from a second set of sensors (a second set of sensors). The method includes dynamically determining, by the electronic device, a position of the object relative to a screen of the electronic device based on data received from a second set of sensors. The method includes determining, by the electronic device, a distance between the object and a screen of the electronic device based on the position of the object. The method includes determining, by the electronic device, whether a distance between the object and a screen of the electronic device meets a distance threshold value. The method may include performing, by the electronic device, an operation based on the position of the object when it is determined that the distance between the object and the screen of the electronic device satisfies the distance threshold value.

An embodiment of the disclosed invention provides an electronic device for controlling an operation using an object. The electronic device includes a memory, a processor, a plurality of sensors, a screen, and an operation control unit, and the operation control unit is connected to the memory and the processor. The motion controller may be configured to detect an object hovered over a screen of the electronic device. The motion control may be configured to obtain data from a first set of sensors in the plurality of sensors. The operation control unit may be configured to determine a distance threshold value of the object to the screen of the electronic device based on data received from the first sensor set. The motion control may be configured to obtain data from a second set of sensors in the plurality of sensors. The motion control unit may be configured to dynamically determine the position of the object with respect to the screen of the electronic device based on data received from the second sensor set. The motion controller may be configured to determine a distance between the object and the screen of the electronic device based on the position of the object. The operation controller may be configured to determine whether a distance between an object of the electronic device and the screen satisfies a distance threshold value. When it is determined that the distance between the object and the screen of the electronic device satisfies the distance threshold, the operation controller may be configured to perform an operation according to the position of the object.

Unlike the existing methods and systems, the proposed method can be used to control the operation of an electronic device using an object such as a ferromagnetic tool (eg, a screwdriver, a spanner, a nail cutter, a metal paper clip, etc.). The electronic device may use a plurality of existing sensors of the electronic device to detect a user input performed through a screen of the electronic device using an object. Thus, this method can be used in electronic devices to incorporate the functionality of a stylus as in traditional touch sensitive devices. Since the stylus is not included in the design of the electronic device, the size, weight, and manufacturing cost of the electronic device can be greatly reduced.

Unlike existing methods and systems, the electronic device is effective in accurately determining the position of an object hovered over a screen of the electronic device using a plurality of existing sensors. The electronic device may consider “hovering on the screen” as a user input such as a gesture, alphabet input, or command for performing an operation. Thus, this method can be used in electronic devices to incorporate the functionality of a stylus as in traditional touch sensitive devices. Since the electronic device does not include an activation digitizer sheet, the power consumption of the electronic device can be greatly improved using the proposed method.

The proposed method enables a user to sign on the screen of the electronic device using an object for authentication, thereby eliminating the need for a fingerprint sensor in the electronic device for authentication. Since the electronic device does not include a fingerprint sensor, the power consumption of the electronic device can be greatly improved using the proposed method, and the manufacturing cost can be greatly reduced.

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

2A is a control block diagram of an electronic device according to an exemplary embodiment.

Referring to FIG. 2A , the electronic device 100 according to an embodiment includes an operation controller 110 , a memory 120 , a processor 130 , a sensor 140 , a screen 150 , a speaker 160 , and a communication unit ( 170) may be included.

The operation control unit 110 may be connected to the memory 120 and the processor 130 , or may be provided integrally with the memory 120 and the processor 130 . For example, the operation control unit 110 may be provided integrally with the processor 130 , and an operation of the operation control unit 110 to be described later may be performed by the processor 130 .

The operation control unit 110 is physically implemented by an analog or digital circuit such as a logic gate, an integrated circuit, a microprocessor, a microcontroller, a memory circuit, a passive electronic device, an active electronic device, an optical device, a hard wire circuit, and the like, and is stored in firmware. can be selectively driven by For example, the circuit may be implemented with one or more semiconductor chips or may be implemented on a substrate support such as a printed circuit board.

The sensor 140 may be provided in plurality, and may include, for example, an accelerometer sensor 141 , a gyro sensor 142 , a microphone 143 , and a magnetometer sensor 144 .

The screen 150 may be an electronic display device such as a liquid crystal display (LCD) screen. However, the type of display panel constituting the screen 150 is not limited.

The speaker 160 may be an electronic device that generates an audible signal using power.

Examples of the operation of the electronic device 100 to be described below include, but are not limited to, alphabet input, user's signature input, drawing generation, user profile start, user's command identification, and gesture identification.

As an example of the user's gesture, shapes such as alphabets, smiles, numbers, and symbols may be formed by moving an object.

The electronic device 100 may be a user equipment (UE), a smart phone, a tablet computer, a personal digital assistance (PDA), a desktop computer, the Internet of things (IoT), etc., and the type is limited. none.

In this case, the object is a device made of a ferromagnetic material such as iron, cobalt, nickel, or an alloy. Examples of objects include, but are not limited to, screwdrivers, nail cutters, metal paper clips, and pens.

The operation controller 110 may be configured to detect an object hovered over the screen 150 of the electronic device 100 .

Specifically, the operation controller 110 may be configured to acquire magnetic field data from the magnetometer sensor 144 among the plurality of sensors 140 of the electronic device 100 . In this case, the magnetometer sensor 144 may acquire magnetic field data in front of the screen 150 .

That is, the operation controller 110 may detect an object hovering over the screen 150 based on the magnetic field data. The operation controller 110 may detect that the object hovers over the screen 150 when it is determined that the rate of change of the magnetic field in front of the screen 150 exceeds a magnetic field threshold based on the magnetic field data.

The magnetic field data is a value of the magnetic field (B) in the (X, Y, Z) direction of a three-dimensional Cartesian coordinate system (ie, the real world). Examples of magnetic field data (0.00 μT, 5.90 μT, -48.40 μT) indicate magnetic fields in the (X, Y, Z) direction, respectively. The operation control unit 110 may be configured to determine whether the magnetic field data meets a magnetic field threshold value. Here, the magnetic field threshold value for each object may be stored in advance in the memory 120 . The magnetic field threshold is the rate at which the magnetic field changes. For example, the speed range is greater than 0.05 micro tesla and less than 0.98 micro tesla. The rate of change of the magnetic field (ie, ΔB/Δt) according to an embodiment is matched with a magnetic field threshold to determine whether the magnetic field data meets the magnetic field threshold. The rate of change of the magnetic field is the change in the magnetic field with time. An example for the rate of change of the magnetic field is 14 micro tesla/sec (ie, 14 μT/s).

Also, when it is determined that the magnetic field data satisfies a magnetic field threshold, the operation control unit 110 may be configured to detect an object hovered over the screen 150 . For example, when the magnetic field data (ie, ΔB/Δt) is greater than the magnetic field threshold, that is, when the magnetic field data meets the magnetic field threshold, the operation control unit 110 determines that the object is hovering over the screen 150 . can be configured to determine. When the magnetic field data (ie, ΔB/Δt) is less than or equal to the magnetic field threshold, that is, when the magnetic field data does not meet the magnetic field threshold, the operation control unit 110 determines that the object does not hover over the screen. can be configured.

The operation control unit 110 may be configured to obtain data from the first set 143 , 144 of the plurality of sensors 140 . A first set 143 , 144 of sensors 140 according to one embodiment includes a microphone 143 and a magnetometer sensor 144 . The motion controller 110 determines a distance threshold value (eg, 62 mm) of the object to the screen 150 of the electronic device 100 based on data received from the first set 143 and 144 of the sensor 140 . can be configured to The distance threshold value means a minimum distance between an object for detecting an object by the electronic device 100 and the screen 150 .

Specifically, the operation controller 110 may determine a distance threshold value corresponding to the object based on the size of the object and the type of the object.

The operation control unit 110 according to an embodiment determines the size (eg, length, width, height, radius, etc.) of the object based on data received from the microphones 143 of the first set 143 and 144 . can be configured to Examples of sizes may include length, width, height, radius, and the like.

The operation control unit 110 controls the speaker 160 to output a sound signal toward the object when an object is detected, and the sound signal output from the speaker 160 and reflected from the object and received by the microphone 143 . The size of the object can be determined based on the data of

Specifically, the operation controller 110 may be configured to transmit a first signal (eg, an audio signal) from the speaker 160 of the electronic device 100 toward the object. Also, the operation controller 110 may be configured to receive a second signal (eg, an audio signal) reflected from the object using the microphone 143 in response to the first signal. Also, the operation controller 110 may be configured to determine the size of the object based on the second signal reflected from the object. The operation control unit 110 according to an embodiment may be configured to determine whether a temporal displacement of the reflected signal (ie, the second signal) is greater than a displacement threshold value. The displacement threshold is pre-stored in the memory 120 . The temporal displacement of the reflected signal is greater than the pre-stored threshold (ie, the displacement threshold) the time it takes to detect the reflected signal after it is emitted. An example of a pre-stored threshold is 1450 µs. The operation control unit 110 may be configured to determine the size of the object when the temporal displacement of the second signal is greater than the displacement threshold value.

The operation controller 110 according to an embodiment may be configured to recognize the type of an object by applying a machine learning (ML) model 119 (see FIG. 2B ) to data received from the magnetometer sensor 144 . An example of the ML model 119 is a recurrent neural network (RNN). The type of object may represent the ferromagnetic material used to make the object.

In other words, the operation controller 110 may determine the type of the object based on the output of the machinability model to which the magnetic field data is input.

The operation controller 110 according to an embodiment may be configured to determine a threshold value of the distance of the object to the screen 150 of the electronic device 100 based on the size of the object and the type of the object. Specifically, the operation controller 110 may be configured to determine the volume of the object by using the size of the object. Also, the operation controller 110 may be configured to determine a threshold value of the distance of the object to the screen 150 of the electronic device 100 based on the volume of the object and the type of the object.

When the distance between the object and the screen 150 is less than the distance threshold, the operation controller 110 according to an embodiment may perform an operation corresponding to the position of the object.

In this case, the operation controller 110 may determine the position of the object based on the object's speed, magnetic field data, and the direction of the electronic device.

Specifically, the operation control unit 110 may be configured to obtain data from the second set 141 , 142 , 144 of the plurality of sensors 140 . The second set of sensors 140 , 141 , 142 , 144 may include an accelerometer sensor 141 , a gyro sensor 142 and a magnetometer sensor 144 .

The operation control unit 110 may be configured to dynamically determine the position of the object with respect to the screen 150 of the electronic device 100 based on data received from the second set 141 , 142 , 144 of the sensor 140 . can

The motion control unit 110 according to an embodiment may be configured to determine the velocity of an object based on accelerometer data received from the accelerometer sensor 141 in the second set 141 , 142 , 144 of the sensor 140 . . The accelerometer data may include acceleration values of the electronic device 100 in the 3D Cartesian coordinate system (X, Y, Z) direction. Examples of the accelerometer data (0.00m/s2, 9.81m/s2, and 0.00m/s2) may indicate the acceleration in the (X, Y, Z) direction of the electronic device 100 , respectively. Examples of the object velocity (-0.2 m/s, 0.4 m/s, -0.14 m/s) may indicate the velocity of the electronic device 100 in the (X, Y, Z) directions, respectively.

The operation controller 110 according to another embodiment may be configured to determine the speed of the object based on a change in magnetic field data generated while the object hovers over the screen 150 (refer to FIG. 8 ).

The operation controller 110 may be configured to identify a direction of the electronic device 100 based on direction data received from the gyro sensor 142 in the second set 141 , 142 , and 144 of the sensor 140 .

The direction of the electronic device 100 may be identified based on azimuth (α), pitch (β), and roll (γ) information of direction data. Examples of azimuth, pitch and roll are -55˚, -28˚, -64˚, respectively.

The operation control unit 110 is configured to continuously determine the position of the object with respect to the screen 150 based on the direction of the electronic device 100 , the speed of the object, and magnetometer data (magnetic field data) received from the magnetometer sensor 144 . can be

The motion control unit 110 according to an embodiment may be configured to determine a position vector of the object with respect to a 3D Cartesian coordinate system based on the object's velocity and magnetometer data. An example of an object's position vector with respect to the (X, Y, Z) direction of a three-dimensional Cartesian coordinate system is (-1.3, 6.7, 4.1), respectively. <Equation 1> for determining the position vector of an object with respect to a three-dimensional Cartesian coordinate system (eg, world plane), that is, r _{(World Plane)} is as follows.

<Equation 1>

Here, B _x , B _y , and B _z mean magnetic field values in the X, Y, and Z directions of the Cartesian coordinate system, respectively.

,

,

is the change in the magnetic field in the X, Y, and Z directions of the Cartesian coordinate system, respectively. Here, the directions v _x , v _y , and v _z mean the velocity of the object in the X, Y, and Z directions of the Cartesian coordinate system, respectively.

Also, the operation controller 110 may be configured to dynamically determine the position of the object with respect to the screen 150 based on the position vector of the object and the direction of the electronic device 100 . <Equation 2> for dynamically determining the position of an object, that is, r _{(Device Plane)} is as follows.

<Equation 2>

here,

to be.

The operation controller 110 may be configured to determine a distance between the object of the electronic device 100 and the screen 150 based on the position of the object. The operation controller 110 may be configured to determine whether a distance between the object of the electronic device 100 and the screen 150 satisfies a distance threshold value.

For example, when the distance between the object of the electronic device 100 and the screen 150 is less than the distance threshold, the operation controller 110 determines that the distance between the object of the electronic device 100 and the screen 150 is the distance It can be determined that the threshold is met.

When the distance between the object of the electronic device 100 and the screen 150 is greater than or equal to the distance threshold, the operation controller 110 determines that the distance between the object of the electronic device 100 and the screen 150 does not satisfy the distance threshold. can decide not to.

When it is determined that the distance between the object of the electronic device 100 and the screen 150 satisfies the distance threshold, the operation controller 110 may perform an operation according to the position of the object. The operation controller 110 may be configured to determine a part (point) of the screen 150 of the electronic device 100 pointed to by the object based on the position of the object. The operation controller 110 may be configured to detect a user interface (UI) data item including a portion of the screen 150 of the electronic device 100 . Examples of UI data items include, but are not limited to, icons, widgets, videos, images, and UIs of applications. The operation control unit 110 may be configured to determine an operation to be performed based on the UI data item. The operation control unit 110 may be configured to perform an operation.

In addition, the operation controller 110 may determine a movement form of the object on the screen 150 based on the position of the object and control the screen 150 to display a character corresponding to the movement form of the object.

Also, when the position of the object moves from the first part of the screen 150 to the second part of the screen 150 , the operation controller 110 may change the user profile of the electronic device 100 to correspond to the first part. The first user profile may be changed to a second user profile corresponding to the second part.

The memory 120 may store data from the plurality of sensors 140 . The memory 120 may store a displacement threshold and a magnetic field threshold. Also, the memory 120 may provide the displacement threshold and the magnetic field threshold to the operation controller 110 in response to receiving a request from the operation controller 110 .

Memory 120 may include a non-volatile storage element. Examples of such non-volatile storage elements may include magnetic hard disks, optical disks, floppy disks, flash memory, or any form of electrical programmable memory (EPROM) or electrical erasable and programmable (EEPROM) memory. Also, memory 120 may be considered a non-transitory storage medium in some examples. The term “non-transitory” may indicate that the storage medium is not embodied as a carrier wave or a propagated signal. However, the term “non-transitory” is not to be interpreted as meaning that the memory 120 is immovable. In some examples, memory 120 may be configured to store larger amounts of information. In certain instances, the non-transitory storage medium may store data that may change over time (eg, random access memory (RAM) or cache).

The processor 130 is configured to execute instructions stored in the memory 120 . The processor 130 is a general-purpose processor such as a central processing unit (CPU), an application processor (AP), and the like, a graphics processing unit (GPU), and a visual processing unit (VPU). It may be a graphics-only processing unit such as, for example. Processor 130 may include multiple cores to execute instructions.

The communication unit 170 may be configured to internally communicate between hardware components of the electronic device 100 . Also, the communication unit 170 may be configured to facilitate communication between the electronic device 100 and other devices. The communication unit 170 may include an electronic circuit specialized for a standard that enables wired or wireless communication.

Although FIG. 2A illustrates various hardware components of the electronic device 100, it should be understood that other embodiments are not limited thereto. In another embodiment, the electronic device 100 may include a smaller number of components. In addition, labels or names of components are used for illustrative purposes only and do not limit the scope of the present invention.

2B is a block diagram of the operation controller 110 for performing an operation of the electronic device 100 according to an exemplary embodiment.

Referring to FIG. 2B , the motion control unit 110 according to an embodiment includes an object detector 111 , a size estimation engine 112 , an object classifier 113 , a distance threshold value estimator 114 , and a speed determination engine 115 . , a direction estimator 116 , a position estimation engine 117 , an action executor 118 , and an ML model 119 .

An object detector 111 , a size estimation engine 112 , an object classifier 113 , a distance threshold value estimator 114 , a velocity determination engine 115 , a direction estimator 116 , and a position estimation engine 117 according to an embodiment ), the action executor 118 and the ML model 119 are for analog or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronics, active electronics, optics, hardwired circuits, and the like. It may be physically implemented by a firmware and selectively driven by firmware. For example, the circuit may be implemented with one or more semiconductor chips or may be implemented on a substrate support such as a printed circuit board.

The object detector 111 may detect an object hovered over the screen 150 of the electronic device 100 . The object detector 111 according to an embodiment may acquire magnetic field data from the magnetometer sensor 144 . The object detector 111 may determine whether the magnetic field data meets a magnetic field threshold. Further, the object detector 111 may detect an object hovered over the screen 150 in response to determining that the magnetic field data meets a magnetic field threshold.

The distance threshold value estimator 114 may obtain data from the first set 143 , 144 of the sensors 140 . The distance threshold value estimator 114 may determine a distance threshold value of the object to the screen 150 based on data received from the first set 143 , 144 of the sensors 140 .

The size estimation engine 112 according to an embodiment determines the size of the object based on data received from the microphone 143 . The size estimation engine 112 according to an embodiment transmits a first signal from the speaker 160 of the electronic device 100 toward the object. In addition, the magnitude estimation engine 112 receives a second signal reflected from the object using the microphone 143 in response to the first signal. Also, the size estimation engine 112 may determine the size of the object based on the second signal reflected from the object. The magnitude estimation engine 112 according to an embodiment may determine whether the temporal displacement of the second signal is greater than a displacement threshold value. The size estimation engine 112 may determine the size of the object when the temporal displacement of the second signal is greater than the displacement threshold value.

The object classifier 113 may recognize the type of object by applying the ML model 119 to the data received from the magnetometer sensor 144 . In addition, the distance threshold value estimator 114 may determine the distance threshold value of the object to the screen 150 according to the size of the object and the type of the object. The distance threshold value estimator 114 according to an embodiment may determine the volume of the object by using the size of the object. In addition, the distance threshold value estimator 114 may determine a distance threshold value of the object to the screen 150 according to the volume of the object and the type of the object.

The position estimation engine 117 dynamically determines the position of the object with respect to the screen 150 of the electronic device 100 based on data received from the second set 141 , 142 , 144 of the sensor 140 . The speed determination engine 115 according to an embodiment determines the speed of the object based on a change in accelerometer data or magnetic field data. The direction estimator 116 may identify the direction of the electronic device 100 based on direction data received from the gyro sensor 142 . The position estimation engine 117 may continuously determine the position of the object with respect to the screen 150 based on the direction of the electronic device 100 , the velocity of the object, and magnetometer data received from the magnetometer sensor 144 . The position estimation engine 117 according to an embodiment may determine the position vector of the object with respect to the 3D Cartesian coordinate system based on the velocity and magnetic field data of the object.

The position estimation engine 117 may dynamically determine the position of the object with respect to the screen 150 based on the position vector of the object and the direction of the electronic device 100 .

The location estimation engine 117 may determine a distance between the object and the screen 150 based on the location of the object. The location estimation engine 117 may determine whether a distance between the object of the electronic device 100 and the screen 150 satisfies a distance threshold value.

When the action executor 118 determines that the distance between the object of the electronic device 100 and the screen 150 satisfies the distance threshold, the action executor 118 performs an action according to the location of the object.

The action executor 118 determines the portion of the screen 150 of the electronic device 100 pointed to by the object based on the position of the object. The action executor 118 detects a UI data item that includes a portion of the screen 150 of the electronic device 100 . The action executor 118 determines the action to perform based on the UI data item and performs the action.

Although FIG. 2B shows hardware components of the operation control unit 110, it should be understood that other embodiments are not limited thereto. In another embodiment, the operation control unit 110 may include a smaller number of components. In addition, labels or names of components are used for illustrative purposes only and do not limit the scope of the present invention. One or more components may be coupled together to perform the same or substantially similar function for performing the operation of the electronic device 100 based on the position of the object.

3 and 4 are flowcharts illustrating a method of controlling an operation of the electronic device 100 according to an exemplary embodiment.

Referring to FIG. 3 , in step 302 , the electronic device 100 may detect an object hovered over the screen 150 . Specifically, the object detector 111 may detect an object hovered over the screen 150 .

In operation 304 , the electronic device 100 may obtain data from the first set 143 and 144 of the sensors 140 . Specifically, the distance threshold value estimator 114 may obtain data from the first set 143 , 144 of the sensors 140 .

In operation 306 , the electronic device 100 may determine a distance threshold value of the object to the screen 150 based on data received from the first set 143 and 144 of the sensor 140 . Specifically, the distance threshold value estimator 114 may determine a distance threshold value of the object to the screen 150 based on data received from the first set 143 , 144 of the sensors 140 .

In operation 308 , the electronic device 100 may obtain data from the second set 141 , 142 , and 144 of the sensor 140 . Specifically, the distance threshold value estimator 114 , the speed determination engine 115 , and the direction estimator 116 may obtain data from the second set 141 , 142 , 144 of the sensors 140 .

In operation 310 , the electronic device 100 may dynamically determine the position of the object with respect to the screen 150 based on data received from the second set 141 , 142 , and 144 of the sensor 140 . Specifically, the position estimation engine 117 may dynamically determine the position of the object relative to the screen 150 based on data received from the second set 141 , 142 , 144 of the sensor 140 .

In operation 312 , the electronic device 100 may determine a distance between the object and the screen 150 based on the position of the object. Specifically, the position estimation engine 117 may determine a distance between the object and the screen 150 based on the position of the object.

In operation 314 , the electronic device 100 may determine whether the distance between the object and the screen 150 satisfies a distance threshold. Specifically, the location estimation engine 117 may determine whether the distance between the object and the screen 150 meets a distance threshold value.

In operation 316 , when the electronic device 100 determines that the distance between the object and the screen 150 satisfies a distance threshold, performing an operation based on the position of the object. Specifically, when the action executor 118 determines that the distance between the object and the screen 150 meets the distance threshold, it causes the action to be performed based on the position of the object.

Various operations, actions, blocks, steps, etc. in the flowcharts above may be performed in the order presented, in a different order, or simultaneously. Also, in some embodiments, some of the actions, acts, steps, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present invention.

Referring to FIG. 4 , in step 401 , the electronic device 100 monitors magnetic field data received from the magnetometer sensor 144 . In operation 402, the electronic device 100 determines whether the rate of change of the magnetic field (ie, ΔB/Δt) is greater than a magnetic field threshold. If the electronic device 100 determines that the rate of change of the magnetic field (ie, ΔB/Δt) is not greater than the magnetic field threshold (No in 402 ), the electronic device 100 continues to monitor the magnetic field data ( 401 ).

In step 403 , when it is determined that the rate of change of the magnetic field (ie, ΔB/Δt) is greater than the magnetic field threshold (YES in 402 ), the electronic device 100 detects an object proximate to the electronic device 100 , and a speaker (160) may be used to transmit a first signal at a frequency of 15 kHz towards the object at time t1.

In operation 404 , the electronic device 100 receives the second signal at t2 using the microphone 143 . Here, the second signal is a signal that collides with the first signal for the object and is reflected from the object.

In operation 405, the electronic device 100 may determine whether the temporal displacement of the second signal is greater than a displacement threshold value. When determining that the time displacement of the second signal is not greater than the displacement threshold (No in 405 ), the electronic device 100 may continue to monitor the magnetic field data ( 401 ).

When determining that the temporal displacement of the second signal is greater than the displacement threshold value (YES in 405), the electronic device 100 determines the size of the object in step 406, and determines the size of the object in step 407 based on the magnetometer data type can be identified.

In operation 408, the electronic device 100 determines a distance threshold value based on the size of the object and the type of the object.

In step 409 , if it is determined that the rate of change of the magnetic field (ie, ΔB/Δt) is greater than the magnetic field threshold (YES in 402 ), the electronic device 100 uses the accelerometer data received from the accelerometer sensor 141 to set the object determine the speed of

In operation 410, the electronic device 100 determines a position vector of the object with respect to the 3D Cartesian coordinate system based on the velocity of the object.

In operation 411 , the electronic device 100 identifies the direction of the electronic device 100 based on the direction data received from the gyro sensor 142 .

In operation 412 , the electronic device 100 dynamically determines the position of the object with respect to the screen 150 based on the position vector of the object and the direction of the electronic device 100 .

In operation 413 , the electronic device 100 may estimate the distance between the object and the screen 150 based on the position of the object with respect to the screen 150 .

In operation 414 , the electronic device 100 may determine whether the distance between the object and the screen 150 is less than a distance threshold value.

When it is determined that the distance between the object and the screen 150 is not smaller than the distance threshold (No in 414 ), the electronic device 100 may continue to monitor the magnetic field data ( 401 ).

In operation 415 , when it is determined that the distance between the object and the screen 150 is smaller than the distance threshold value (Yes in 414 ), the electronic device 100 may perform an operation based on the position of the object.

Various operations, actions, blocks, steps, etc. in the flowcharts above may be performed in the order presented, in a different order, or simultaneously. Further, in some embodiments, some of an action, action, block, step, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present invention.

5A is a diagram illustrating a signal received by the microphone 143 of the electronic device 100 according to an exemplary embodiment.

The amplitude 501 of the second signal received at the microphone 143 at various times 502 is shown in FIG. 5A . Here, the second signal is a signal reflected by the first signal output from the speaker 160 to the object hitting the object.

The electronic device 100 does not know an appropriate second signal to be processed in order to estimate the distance between the object and the electronic device 100 . In order to determine an appropriate second signal, the user must move the object back and forth in a direction parallel to the screen 150 of the electronic device 100 . The electronic device 100 may determine the relevant movement time displacement of the appropriate second signal based on the movement of the object, and the unrelated second signal may be maintained close to the specific movement time. The related movement time displacement is a displacement indicating whether the object moves away from or approaches the electronic device 100 . The movement time is the time it takes for the signal output from the speaker 160 to travel to the microphone 143 after being reflected from the object. The electronic device 100 may determine a plurality of accumulated detection and dispersion of the second signal, that is, the standard deviation of the movement time of each second signal. The electronic device 100 may select a second signal exceeding the variance threshold as an appropriate second signal. The dispersion threshold ranges from 1350 to 1550 μs.

FIG. 5B is a diagram illustrating a case in which the electronic device 100 determines the size of an object hovered over the screen 150 according to an exemplary embodiment.

A side view of the object 510 hovered over the electronic device 100 is illustrated in FIG. 5B . The microphone 143 and the speaker 160 are disposed with a distance 511 from the electronic device 100 . The first signal 512A is transmitted from the speaker 160 and collides with the object 510 . The first signal 512A reflected from the object 510 is the second signal 512B. Also, the microphone 143 may receive the second signal 512B.

Similarly, the first signal 513A is transmitted from the speaker 160 and strikes the object 510 . The first signal 513A reflected from the object 510 is the second signal 513B. Also, the microphone 143 may receive the second signal 513B. Here, the second signal 513B is bent due to the structural characteristics of the object 510 .

The electronic device 100 measures the distance 'x' 511 between the microphone 143 and the speaker 160 using <Equation 3>. Here, the movement time of the signal may be determined by the electronic device 100 .

In operation 410, the electronic device 100 determines a position vector of the object with respect to the 3D Cartesian coordinate system based on the velocity of the object.

In operation 411 , the electronic device 100 identifies the direction of the electronic device 100 based on the direction data received from the gyro sensor 142 .

In operation 412 , the electronic device 100 dynamically determines the position of the object with respect to the screen 150 based on the position vector of the object and the direction of the electronic device 100 .

In operation 413 , the electronic device 100 may estimate the distance between the object and the screen 150 based on the position of the object with respect to the screen 150 .

In operation 414 , the electronic device 100 may determine whether the distance between the object and the screen 150 is less than a distance threshold value.

When it is determined that the distance between the object and the screen 150 is not smaller than the distance threshold (No in 414 ), the electronic device 100 may continue to monitor the magnetic field data ( 401 ).

In operation 415 , when it is determined that the distance between the object and the screen 150 is smaller than the distance threshold value (Yes in 414 ), the electronic device 100 may perform an operation based on the position of the object.

Various operations, actions, blocks, steps, etc. in the flowcharts above may be performed in the order presented, in a different order, or simultaneously. Also, in some embodiments, some of an action, action, block, step, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present invention.

5A is a diagram illustrating a signal received by the microphone 143 of the electronic device 100 according to an exemplary embodiment.

The amplitude 501 of the second signal received at the microphone 143 at various times 502 is shown in FIG. 5A . Here, the second signal is a signal reflected by the first signal output from the speaker 160 to the object hitting the object.

The electronic device 100 does not know an appropriate second signal to be processed in order to estimate the distance between the object and the electronic device 100 . In order to determine an appropriate second signal, the user must move the object back and forth in a direction parallel to the screen 150 of the electronic device 100 . The electronic device 100 may determine the relevant movement time displacement of the appropriate second signal based on the movement of the object, and the unrelated second signal may be maintained close to the specific movement time. The related movement time displacement is a displacement indicating whether the object moves away from or approaches the electronic device 100 . The movement time is the time it takes for the signal output from the speaker 160 to travel to the microphone 143 after being reflected from the object. The electronic device 100 may determine a plurality of accumulated detection and dispersion of the second signal, that is, the standard deviation of the movement time of each second signal. The electronic device 100 may select a second signal exceeding the variance threshold as an appropriate second signal. The dispersion threshold ranges from 1350 to 1550 μs.

FIG. 5B is a diagram illustrating a case in which the electronic device 100 determines the size of an object hovered over the screen 150 according to an exemplary embodiment.

A side view of the object 510 hovered over the electronic device 100 is illustrated in FIG. 5B . The microphone 143 and the speaker 160 are disposed with a distance 511 from the electronic device 100 . The first signal 512A is transmitted from the speaker 160 and collides with the object 510 . The first signal 512A reflected from the object 510 is the second signal 512B. Also, the microphone 143 may receive the second signal 512B.

Similarly, the first signal 513A is transmitted from the speaker 160 and strikes the object 510 . The first signal 513A reflected from the object 510 is the second signal 513B. Also, the microphone 143 may receive the second signal 513B. Here, the second signal 513B is bent due to the structural characteristics of the object 510 .

The electronic device 100 measures the distance 'x' 511 between the microphone 143 and the speaker 160 using <Equation 3>. Here, the movement time of the signal may be determined by the electronic device 100 .

<Equation 3>

Here, the speed of sound, v _sound , is 34300 cm/sec.

Also, the electronic device 100 may determine the size of the object by applying the distance 511 between the microphone 143 and the speaker 160 to the Fresnel-Kirchhoff diffraction equation.

6A and 6B are diagrams illustrating a recurrent neural network (RNN) used by an electronic device to identify a type of an object hovered over a screen, according to an exemplary embodiment.

Table 1 shows absolute values of the magnetic field B measured every second by hovering each type of object on the electronic device 100 according to an embodiment.

object category	B in 1 st second (μT)	B in 2nd second (μT)	B in 3rd second (μT)	B in 4th second (μT)
steel	34.42	35.12	35.67	36.21
nickel	23.32	23.67	23.91	24.12
cobalt	28.56	28.64	28.79	28.86

The change rate (ΔB/Δt) of the magnetic field according to an embodiment is shown in <Table 2>.

object category	△B/△t for 1 st instant (μT)	△B/△t for 2nd instant (μT)	△B/ △t for 3rd instant (μT)
steel	35.12-34.42 = 0.70	35.67-35.12 = 0.55	36.21- 35.67= 0.54
nickel	23.67-23.32 = 0.35	23.91- 23.67 = 0.24	24.12 - 23.91 = 0.21
cobalt	28.64-28.56 = 0.08	28.79-28.64 = 0.15	28.86- 28.79 = 0.07

The electronic device 100 determines the type of object by learning the rate of change of the magnetic field. Hundreds of values of the rate of change of the magnetic field are sufficient to learn the electronic device 100 . Also, the electronic device 100 identifies the type of object based on the change rate of the magnetic field using the learned change rate of the magnetic field. As shown in FIG. 6A , the electronic device 100 applies a rate of change of the magnetic field to the RNN 600 to identify the type of object. X ₁ , X ₂ , X ₃ are inputs of the RNN 600 . X ₁ is ΔB/Δt for the first instant. X ₂ is ΔB/Δt for the second instant. X ₃ is ΔB/Δt for the third. f _w (602), f _w (604), f _w (606) are functions with parameter 'W' (f _w ). h ₀ (601), h ₁ (603), h ₂ (605), h ₃ (607), ... h _t (609) may be defined as in <Equation 4>.

<Equation 4>

where h _t-1 is the old output value and h _t is the new output value. <Equation 4> can also be expressed as follows.

Here, W(608) means a weight matrix (W _hh ) used in every instantaneous function f _w . The weight matrix according to an embodiment may be obtained by training using a backpropagation method. W _y (610) means the weight matrix used by h _t (609) to generate the output Y (611). Here, Y(611) denotes the output of the RNN 600 given in <Equation 5> indicating the type of object for the given inputs X ₁ , X ₂ , and X ₃ .

<Equation 5>

In an example, it may be assumed that X ₁ , X ₂ , and X ₃ are 0.70, 0.55, and 0.54, respectively. In this case, the electronic device 100 may identify iron as the type of object. A flowchart of a method of identifying a type of an object using the RNN 600 is shown in FIG. 6B . The input of the RNN 600 is X (612), where X (612) means the rate of change of the magnetic field at each instant. The output of the RNN 600 is Y 611 , where Y 611 means the type of object such as iron, cobalt, nickel, or the like.

7 is a diagram illustrating a plot of a volume of an object with respect to a height of the object from the screen 150 of the electronic device 100 according to an exemplary embodiment.

Referring to FIG. 7 , the electronic device 100 may determine the volume of the object based on the size of the object. Also, the electronic device 100 may determine the distance threshold value of the object by performing regression analysis on the volume of the object. The electronic device 100 uses regression analysis to estimate the value of the dependent variable based on the value of the at least one independent variable, where the dependent variable is the height of the object from the screen 150 and the independent variable is the height of the object. is the volume

It is considered that the volume plot on the graph in the X direction and the height of the object from the screen 150 are plotted on the graph in the Y direction with the same relation as in Equation (6). The relationship between volume and height can be described by a linear function 707 . That is, it is assumed that the change in height is due to the change in volume. 705 denotes a point plotted on the graph according to the relationship between volume and height.

<Equation 6>

where Y _i 702 is the expected height of the object from the screen 150 relative to the object's volume 'X _i ' 706 , while the actual value of Y i.e. the screen 150 of X _i relative to the volume 706 . ), the actual height of the object is 701. β ₀ (703) and β ₁ (708) represent constants determined according to the type of object. β ₀ and β ₁ are trainable parameters whose values are determined using <Equation 8> and <Equation 9>. The values of β ₀ (703) and β ₁ (708) are real numbers. ε _i (704) represents an arbitrary error range between 10 ^-2 and 10 ^-4 . The random error can be calculated using the minimization of <Equation 6> using gradient descent. Also, <Equation 6> may be modified to <Equation 7> according to random error calculation.

<Equation 7>

After determining the volume, the electronic device 100 determines the height at which the object is sensed. X is the volume and Y is the height. The electronic device 100 uses <Equation 8> and <Equation 9> to calculate b ₀ and b ₁ by minimizing the loss between the predicted value and the actual value using gradient descent.

<Equation 8>

<Equation 9>

In one example, distance thresholds for different objects for different volumes are as provided in Table 3.

object type	Volume (cm 2 )	Depth (mm)
steel	35.90	62
nickel	29.60	46
cobalt	54.70	56
steel	45.65	78
nickel	69.55	64
cobalt	79.50	68
steel	60.50	92
nickel	24.58	40
cobalt	80.65	70

8 is a diagram illustrating a case in which an electronic device determines a speed of an object hovered over a screen according to an exemplary embodiment.

Referring to FIG. 8 , the electronic device 100 may learn to determine the speed of the object 510 , for example, a screw driver. During the learning phase, the object 510 should be placed on the screen 150 of the electronic device 100 . In addition, the electronic device 100 must move in the horizontal and vertical directions 801 - 804 by fixing the position of the object 510 .

The electronic device 100 may monitor magnetic field data and accelerometer data at every instant while moving the electronic device 100 . The electronic device 100 may determine the magnetic field at every moment by using the accelerometer data. The electronic device 100 may determine the speed of the electronic device 100 at every moment based on the accelerometer data. Examples of magnetic field and velocity are shown in <Table 4>. Here, B _t-1 is the magnetic field at time 0 second and B _t represents the magnetic field at time 1 second. Accordingly, the electronic device 100 may learn the relationship between the speed of the electronic device 100 and the magnetic field.

B t-1 (μT)	B t (μT)	speed (m/s 2 )
0.70,5.4,3.6	0.6,4.9,2.9	-0.2,0.4,-0.14
0.35,0.21,0.27	0.24,0.31,0.11	0.21,0.12,-0.51
0.08,0.82,0.33	0.72,0.15,0.91	1.23,-0.98,0.87

Also, when the object 510 hovers over the screen 150 , the electronic device 100 determines the speed of the object 510 by using the learned knowledge about the relationship between the speed of the electronic device 100 and the magnetic field. .

The electronic device 100 determines the speed of the object 510 by performing regression analysis on the magnetic field. The electronic device 100 may use regression analysis to estimate the value of the dependent variable based on the value of the at least one independent variable. Here, the dependent variable is the speed of the object 510 and the independent variable is a change in the magnetic field due to the movement of the object 510 . The relationship between the velocity of the object 510 and the change in the magnetic field may be described as a linear function given in Equation (6).

Y _i is the estimated speed of the object 510 according to the change of the magnetic field 'X _i '. β ₀ and β ₁ are constants determined according to the type of the object 510 . β ₀ and β ₁ are trainable parameters whose values are determined using <Equation 8> and <Equation 9>. The values of β ₀ and β ₁ are real numbers. ε _i is a random error range between 10 ^-2 and 10 ^-4 . An arbitrary error can be calculated using the minimization of <Equation 6> using the gradient descent method. After confirming the change in the magnetic field, the electronic device 100 may determine the speed. X is the change in the magnetic field and Y is the velocity. The electronic device 100 may use <Equation 8> and <Equation 9> to calculate b ₀ and b ₁ by minimizing the loss between the predicted value and the actual value using gradient descent.

In an example, the electronic device 100 determines the position vector of the object 510 based on the speed of the object 510 . Magnetic field 'B _t-1 ' at time 0 seconds, magnetic field 'B _t ' at time 1 second, change of magnetic field 'ΔB', corresponding velocity of object 510 and corresponding position vector of object 510 'r _{(World Plane)} ' is shown in Table 5.

B t (μT)	B t-1 (μT)	ΔB (μT)	speed (m/s)	r(World Plane)
0.6,4.9,2.9	0.70,5.4,3.6	0.1,0.5,0.7	-0.2,0.4,-0.14	-1.3,6.7,4.1
0.24,0.31,0.11	0.35,0.21,0.27	0.11,-0.10,0.16	0.21,0.12,-0.51	2.1,5.6,2.6
0.72,0.15,0.91	0.08,0.82,0.33	-0.64,0.67,-0.58	1.23,-0.98,0.87	-1.3, -0.6,0.3

9 is a diagram illustrating a method of calculating a direction of the electronic device 100 according to an embodiment, and FIG. 10 is a position of an object 510 in a 3D Cartesian coordinate system in the electronic device 100 according to an embodiment. It is a diagram showing the position of the object 510 with respect to the vector and the screen 150 .

Referring to FIG. 9 , X ( 901 ), Y ( 902 ), and Z ( 903 ) are axes of a three-dimensional Cartesian coordinate system in a positive direction. The electronic device 100 is arranged such that the screen 150 faces the positive Z-axis 903 and is disposed at the center of the X-axis 901, the Y-axis 902, and the Z-axis 903, and the positive Y-axis ( The upper end of the electronic device 100 may be disposed in a direction toward 902 . Then the azimuth 'α' 906 is the angle between the positive Y-axis 902 and magnetic north 907, and the azimuth 906 ranges from 0 degrees to 360 degrees. A positive roll 'γ' 905 is defined as the electronic device 100 starts lying flat on a horizontal plane and the positive Z-axis 903 begins to tilt toward the positive X-axis 901 . The positive pitch 'β' 904 is defined as a case in which the electronic device 100 starts lying flat on a horizontal plane and the positive Z-axis 903 starts to tilt toward the positive Y-axis 902 .

Referring to FIG. 10 , the object 510 may be positioned on the electronic device 100 with position coordinates (x3, y3, z3) 1005 and a height 'h' 1004 above the screen 150 . In this case, the height 'h' 1004 at which the object 510 is positioned from the screen 150 may be lower than the distance threshold value 805 . Also, the position coordinates of the magnetometer sensor 144 of the electronic device 100 may be (x1, y1) (1002).

The electronic device 100 may determine the position vector of the object 510 as r(1001). Also, the electronic device 100 uses the position vector of the object 510 and the direction of the electronic device 100 to determine the position (ie, position coordinates) of the object 510 with respect to the screen 150 or (x2, y2). ) 1003 may determine an arbitrary plane of the electronic device 100 .

11 is a diagram illustrating an exemplary scenario in which the electronic device 100 receives a text based on the position of the object 510 according to an embodiment.

Referring to FIG. 11 , as shown in 1100A, a user uses a screw driver 510 , ie, an object 510 on the screen 150 of the electronic device 100 , in the shape of an 'S' 1101 , a distance threshold value. You can hover below (805).

In this case, the electronic device 100 determines the position of the screw driver 510 at every moment the screw driver 510 hovers over the screen 150 . Also, the electronic device 100 may sense a movement form of the object 510 on the screen 150 at every moment based on the position of the screw driver 510 . The electronic device 100 identifies the movement form of the object 510 as 'S' and takes the input as the alphabet 'S'. Also, as shown in 1100B, the electronic device 100 displays the alphabet 'S' 1102 to the user.

Therefore, the proposed method can be used in the electronic device 100 to integrate the function of a stylus as in a traditional touch sensing device. By not including the stylus in the design of the electronic device 100 , the size, weight, and manufacturing cost of the electronic device 100 may be greatly reduced. In addition, since the electronic device 100 does not include an active digitizer sheet, power consumption of the electronic device 100 can be greatly improved by using the proposed method. The proposed method allows a user to sign the screen 150 of the electronic device 100 using an object to be authenticated, so that the electronic device 100 does not need a fingerprint sensor for authentication. By not including the fingerprint sensor in the electronic device 100 , power consumption of the electronic device 100 can be greatly improved and manufacturing cost can be greatly reduced using the proposed method.

12 is a diagram illustrating an exemplary scenario in which the electronic device 100 switches a user profile based on a location of an object 510 according to an embodiment.

Referring to FIG. 12 , the electronic device 100 sets a boundary 1201 between the first part 1202 and the second part 1203 to display the UI of the electronic device 100 with the first part 1202 and the second part 1202 . Divide into two parts 1203.

The electronic device 100 may allocate the first part 1202 as a part for the first user profile and allocate the second part 1203 as a part for the second user profile.

The electronic device 100 may activate the first user profile when the user hovers the object into the first portion 1202 on the screen 150 within the distance threshold value range. Also, the electronic device 100 may activate the second user profile when the user hovers the object into the second portion 1203 on the screen 150 within the distance threshold range.

Accordingly, this method enables the electronic device 100 to switch a user profile based on a portion in which the user hovers an object.

13 is a diagram illustrating an exemplary scenario in which the electronic device 100 instructs the user to wash the user's hand based on the position of the object, according to an embodiment.

Referring to FIG. 13 , a user may wear a metal ring 1304 that is an object. Also, as shown in 1300 , the user may wash hands 1303 using water 1302 flowing from the faucet 1301 .

The electronic device 100 according to an embodiment may monitor the position of the metal ring 1304 while washing the hands 1303 when the position of the metal ring 1304 is within a distance threshold value range. That is, the electronic device 100 may track the movement of the metal ring 1304 while the user washes the hand 1303 .

In response to detecting the movement of the metal ring 1304 , the electronic device 100 may start and track a timer, as shown at 1305 . Also, according to the movement of the metal ring 1304 , the electronic device 100 may track whether the user is washing their hands for up to 20 seconds. As illustrated in 1306 , the electronic device 100 may instruct the user to continue hand washing when the movement of the metal ring 1304 is not detected within 20 seconds after the washing starts. The timer expires after 20 seconds, as shown in 1307 , and the electronic device 100 may instruct the user to stop washing the hands 1303 . Accordingly, the electronic device 100 of the present invention can intelligently track the tasks performed by the user to improve the user experience and provide the best recommendations to the user.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create a program module to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

The computer-readable recording medium includes any type of recording medium in which computer-readable instructions are stored. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage, and the like.

The disclosed embodiments have been described with reference to the accompanying drawings as described above. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be practiced in other forms than the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims (15)

1. In an electronic device,

   screen;

   a magnetometer sensor for acquiring magnetic field data in front of the screen; and

   processor; including;

   The processor is

   When detecting an object hovering over the screen based on the magnetic field data, determine a distance threshold value corresponding to the object based on the size and type of the object,

   When the distance between the object and the screen is less than the distance threshold, the electronic device performs an operation corresponding to the position of the object.
2. According to claim 1,

   The processor is

   and detecting that the object hovers over the screen when it is determined based on the magnetic field data that the rate of change of the magnetic field in front of the screen exceeds a magnetic field threshold.
3. According to claim 1,

   The electronic device is

   speaker; and

   It further comprises a microphone;

   The processor is

   When detecting the object, the speaker is controlled to output a sound signal toward the object, and the size of the object is determined based on the data of the sound signal reflected from the object and received by the microphone after being output from the speaker Determining electronic device.
4. According to claim 1,

   The processor is

   An electronic device that determines the type of the object based on an output of a machine learning (ML) model to which the magnetic field data is input.
5. According to claim 1,

   The processor is

   An electronic device to determine the position of the object based on the speed of the object, the magnetic field data, and a direction of the electronic device.
6. 6. The method of claim 5,

   The electronic device is

   accelerometer sensor; and

   Gyro sensor; further comprising,

   The processor is

   The electronic device determines the speed of the object based on the accelerometer data output from the accelerometer sensor, and determines the direction of the electronic device based on the direction data output from the gyro sensor.
7. 7. The method of claim 6,

   The processor is

   Determine a position vector of the object with respect to a three-dimensional Cartesian coordinate system based on the speed of the object and the magnetic field data, and determine the position of the object with respect to the screen based on the position vector of the object and the direction of the electronic device electronic device.
8. According to claim 1,

   The processor is

   The electronic device determines a point pointed to by the object on the screen based on the position of the object, determines a user interface data item including the point, and performs an operation corresponding to the user interface data item.
9. According to claim 1,

   The processor is

   An electronic device that determines a movement form of the object on the screen based on the position of the object, and controls the screen to display a character corresponding to the movement form of the object.
10. According to claim 1,

    The processor is

    When the position of the object moves from the first part of the screen to the second part of the screen, the user profile of the electronic device is changed from the first user profile corresponding to the first part to the second part corresponding to the second part. 2 Electronic device changing to user profile.
11. A method for controlling an electronic device comprising a screen and a magnetometer sensor for acquiring magnetic field data in front of the screen,

    when detecting an object hovering over the screen based on the magnetic field data, determine a distance threshold value corresponding to the object based on the size and type of the object;

    and performing an operation corresponding to the position of the object when the distance between the object and the screen is less than the distance threshold.
12. 12. The method of claim 11,

    Detecting the object is

    and detecting that the object hovers over the screen when it is determined that the rate of change of the magnetic field in front of the screen exceeds a magnetic field threshold based on the magnetic field data.
13. 12. The method of claim 11,

    The electronic device is

    speaker; and

    It further comprises a microphone;

    when detecting the object, control the speaker to output a sound signal toward the object;

    and determining the size of the object based on data of an acoustic signal output from the speaker and reflected from the object and received by the microphone.
14. 12. The method of claim 11,

    and determining the type of the object based on an output of a machine learning (ML) model to which the magnetic field data is input.
15. 12. The method of claim 11,

    and determining the position of the object based on the speed of the object, the magnetic field data, and the direction of the electronic device.

Images