WO2022095915A1

WO2022095915A1 - Electronic device, method and storage medium

Info

Publication number: WO2022095915A1
Application number: PCT/CN2021/128556
Authority: WO
Inventors: 闫冬生
Original assignee: 索尼半导体解决方案公司; 闫冬生
Priority date: 2020-11-04
Filing date: 2021-11-04
Publication date: 2022-05-12
Also published as: CN116391163A; CN114449069A

Abstract

The present disclosure relates to an electronic device, a method and a storage medium. An electronic device for a hand-held device, comprising a processing circuit, the processing circuit being configured to: acquire distance data associated with one or more physical features of a user of the hand-held device; determine a direction of the hand-held device relative to the user on the basis of the distance data, so as to determine the hand of the user currently operating the hand-held device; and present, according to a determination result, a user interface (UI) suitable for the operation by the hand of the user.

Description

Electronic device, method and storage medium

This application claims priority to Chinese Patent Application No. 202011216016.8, filed on Nov. 4, 2020, entitled "Electronic Device, Method, and Storage Medium," the disclosure of which is hereby incorporated herein in its entirety.

technical field

The present disclosure generally relates to handheld user equipment such as smart phones, and more particularly, to a method of determining using hands of a user that is operating the handheld device.

Background technique

In recent years, handheld user devices such as smartphones have become increasingly popular. Such handheld devices are usually equipped with a touch screen, which, in addition to the purpose of visual presentation, can also provide interaction with the user by means of a UI containing virtual operating components (such as virtual keys) displayed on the screen, thereby reducing the need for physical keys. use. However, in order to provide better operation and visual experience, the screen size of handheld devices is getting larger and larger, which virtually increases the difficulty for users to operate the device while holding the device with one hand. Although it is possible to optimize the UI design so that the user's hand can easily complete each operation on the screen, it is necessary to determine whether the user is currently operating the device with the left hand or the right hand, so as to present the UI in a form suitable for operation.

Various methods for judging the user's hand have been proposed. For example, Patent Document 1 (CN105468269A) discloses using proximity sensors 201-209 and 201'-209' mounted on the mobile phone to detect which hand of the user is holding the mobile phone, as shown in Fig. 1A. Patent Document 2 (JP2012023554A) discloses that a sensor such as a gyroscope or an accelerometer is used to detect and judge which hand of the user is holding the mobile phone, as shown in FIG. 1B . However, the gyroscope or accelerometer is affected by the motion state or gravity (vertical direction), and the effect of this judgment method is not good when the user is walking or lying down, so its application scenarios are limited. In addition, Patent Document 3 (CN108958603A) discloses that a common RGB camera is used to capture the user's face, and the user's hand is judged according to the different areas of the left and right faces. But RGB sensors are limited by luminance conditions and are not accurate because the area ratios are ambiguous under different head poses.

Therefore, there is a need for a wider range of applications and a high degree of accuracy in determining which hand a user is operating a device with.

SUMMARY OF THE INVENTION

This section presents a brief summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. It should be understood, however, that this summary is not an exhaustive overview of the present disclosure. It is not intended to identify key or critical parts of the disclosure nor to limit the scope of the disclosure. Its sole purpose is to present some concepts related to the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

According to one aspect of the present disclosure, there is provided an electronic device for a handheld device including processing circuitry configured to obtain a distance associated with one or more physical characteristics of a user of the handheld device determining the direction of the handheld device relative to the user based on the distance data; determining the hand of the user currently operating the handheld device based on the determined method; The user's hand-operated user interface (UI).

According to another aspect of the present disclosure, there is provided a handheld device comprising: a depth camera for obtaining a depth image about a user of the handheld device; and a processing circuit configured to: based on the depth image, obtaining distance data associated with one or more physical features of the user; determining a direction of the handheld device relative to the user based on the distance data; determining that the user is currently operating the user based on the determined direction The user's hand of the handheld device; according to the judgment result, a user interface (UI) suitable for the user's operation by the user's hand is presented.

According to another aspect of the present disclosure, there is provided a method for a handheld device, comprising: obtaining distance data associated with one or more physical characteristics of a user of the handheld device; determining based on the distance data The direction of the handheld device relative to the user; judging the hand of the user currently operating the handheld device based on the determined direction; User Interface (UI).

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing executable instructions that, when executed, implement the above-described method.

By applying one or more aspects of the present disclosure, it is possible to conveniently and accurately determine the user's hand for operating the handheld device and present a UI suitable for the user's operation.

Description of drawings

The present disclosure may be better understood by reference to the following detailed description taken in conjunction with the accompanying drawings, wherein the same or like reference numerals are used throughout the drawings to refer to the same or like elements. All accompanying drawings, which are incorporated in and constitute a part of this specification together with the following detailed description, serve to further illustrate embodiments of the disclosure and to explain the principles and advantages of the disclosure. in:

1A-1B show schematic diagrams of judging the use of hands in the prior art;

FIG. 2 shows the hardware configuration of a smartphone as an example of a handheld device;

3A shows a block diagram of an electronic device according to the present disclosure;

3B shows a flowchart of a method according to the present disclosure;

4A-4B illustrate an example of determining the orientation of a handheld device relative to a user;

5A-5B illustrate another example of determining the orientation of a handheld device relative to a user;

6A-6B illustrate another example of determining the orientation of a handheld device relative to a user.

Detailed ways

Various exemplary embodiments of the present disclosure will hereinafter be described in detail with reference to the accompanying drawings. In the interest of clarity and conciseness, not all implementations of embodiments are described in this specification. It should be noted, however, that many implementation-specific settings may be made according to particular needs in implementing embodiments of the present disclosure.

In addition, it should also be noted that, in order to avoid obscuring the present disclosure due to unnecessary details, only processing steps and/or device structures closely related to the technical solutions of the present disclosure are shown in the drawings. The following description of exemplary embodiments is illustrative only and is not intended to limit the present disclosure and its applications in any way.

FIG. 2 is a block diagram showing a hardware configuration of a smartphone 1600 as an example of the handheld device of the present disclosure. It should be understood that although the present disclosure is described by taking a smartphone as an example, the handheld device to which the technical contents of the present disclosure can be applied is not limited to a smartphone, but can be implemented as various types of mobile terminals, such as tablet computers, personal Computers (PCs), Palmtops, Smart Assistants, Portable Game Terminals, Portable Digital Cameras, and the like.

Smartphone 1600 includes processor 1601, memory 1602, storage device 1603, external connection interface 1604, RGB camera 1605, depth camera 1606, sensor 1607, microphone 1608, input device 1609, display device 1610, speaker 1611, wireless communication subsystem 1612 , bus 1617 , battery 1618 , etc., these components are connected to each other through bus 1617 . The battery 1618 provides power to the various components of the smartphone 1600 via a feeder line (not shown in FIG. 2).

The processor 1601 may be implemented as, for example, a CPU or a system on a chip (SoC), and generally controls the functions of the smartphone 1600 . The processor 1601 may include various implementations of digital circuitry, analog circuitry, or mixed-signal (combination of analog and digital) circuitry, such as integrated circuits (ICs), application specific integrated circuits (ASICs), to perform functions in a computing system ), portions or circuits of a separate processor core, an entire processor core, a separate processor, a programmable hardware device such as a field programmable gate array (FPGA), and/or a system including multiple processors .

The memory 1602 is used to store data and programs executed by the processor 1601, and may be, for example, volatile memory and/or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), Static random access memory (SRAM), read only memory (ROM), flash memory, etc. The storage device 1603 may include a storage medium such as a semiconductor memory and a hard disk in addition to the memory 1602 . The external connection interface 1604 is an interface for connecting external devices such as memory cards and Universal Serial Bus (USB) devices to the smartphone 1600 .

Input device 1609 includes, for example, a keyboard, keypad, keys, or switches for receiving operations or information input from a user. The display device 1610 includes a screen such as a liquid crystal display (LCD) and an organic light emitting diode (OLED) display, and displays an output image of the smartphone 1600 such as a UI including virtual operation parts. Typically, the display device 1610 may be implemented as a touch screen that includes a touch sensor configured to detect a touch on the screen of the display device 1610, whereby the touch screen acts as both a display device and an input device.

The microphone 1608 converts the sound input to the smartphone 1600 into an audio signal. Using voice recognition technology, instructions from the user can be input into the smartphone 1600 in the form of speech through the microphone 1608, whereby the microphone 1608 can also act as an input device. The speaker 1611 converts the audio signal output from the smartphone 1600 into sound.

The wireless communication subsystem 1612 is used to perform wireless communication. The wireless communication subsystem 1612 may support any cellular communication scheme (such as 4G LTE or 5G NR, etc.), or other types of wireless communication schemes, such as short-range wireless communication schemes, near field communication schemes, and wireless local area network (LAN) schemes. Wireless communication subsystem 1612 may include, for example, a BB processor and RF circuitry, antenna switches, antennas, and the like.

The smartphone 1600 also includes an RGB camera 1605 for capturing images of the outside world. The RGB camera 1605 includes optical components and image sensors such as charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors. The light passing through the optics is captured and photoelectrically converted by an image sensor, and processed by, for example, processor 1601 to generate an image. Smartphone 1600 may include more than one RGB camera 1605, such as multiple cameras on the front or rear. The RGB camera 1605 can be composed of a wide-angle camera module, an ultra-wide-angle camera module, a telephoto camera module, etc., to provide excellent shooting effects.

In addition to the conventional RGB camera 1605, the smartphone 1600 may also include a depth camera 1606 for obtaining depth information. Depth cameras are sometimes referred to as 3D cameras. As the name suggests, depth cameras are used to detect the depth of field distance in the shooting space. They are increasingly used in applications such as object recognition, behavior recognition, and scene modeling. Compared with the traditional camera, the depth camera adds a depth measurement function, which makes it more convenient and accurate to perceive the surrounding environment and changes. Depending on the technology utilized, the depth camera 1606 may include, but is not limited to, a Time of Flight (TOF) camera, a Structured Light (Structured Light) camera, a Binocular Stereo Vision camera, and the like. The following is a brief introduction to these depth cameras.

TOF cameras use active detection methods to obtain distances by measuring the time of flight of light. Specifically, a light transmitter (such as an LED or laser diode) continuously emits modulated light pulses, generally invisible infrared light, to the target. sensor) to receive the reflected light. Because of the speed of light, it is not practical to measure the time of flight of light directly, and it is generally realized by detecting the phase shift of the light wave modulated by a certain means. TOF technology can be generally divided into two types according to different modulation methods: pulsed modulation (Pulsed Modulation) and continuous wave modulation (Continuous Wave Modulation). Because the speed of light and the wavelength of the modulated light are known, the arithmetic unit can quickly and accurately calculate the depth distance to the object by using the phase shift between the emitted pulsed light and the received pulsed light. The TOF camera can simultaneously obtain the depth information of the entire image, that is, the two-dimensional depth point cloud information, and the value of each point on the image represents the value of the distance between the camera and the object (ie, the depth value), instead of Like RGB cameras are light intensity values. The main advantages of TOF cameras are: 1) The detection distance is long. In the case of enough laser energy, it can reach several tens of meters. 2) Less interference from ambient light. However, TOF technology also has some obvious problems, such as high requirements for equipment, especially the time measurement module; large computational resource consumption, multiple sampling and integration are required to detect phase offset, and the computational load is large; limited to resource consumption and filtering, frame rate There is no way to achieve a higher resolution.

Structured light cameras also use active detection methods. The principle is to project light with certain structural features (such as encoded images or pseudo-random speckle spots) onto the object to be photographed through a light emitter (such as a near-infrared laser). The emitted structured light pattern is then captured by a special light receiver. This kind of light with a certain structure will capture different image phase information due to different depth regions of the photographed object, and then the change of this structure is converted into depth information by the arithmetic unit according to the principle of triangulation. Compared with TOF cameras, structured light cameras have less computational complexity, lower power consumption, and higher accuracy at close range, so they have great advantages in face recognition and gesture recognition, but they are easily disturbed by ambient light and have poor outdoor experience. , and as the detection distance increases, the accuracy will deteriorate.

The binocular stereo camera adopts passive detection method. Binocular stereo vision is an important form of machine vision. It uses imaging devices (such as traditional RGB cameras) to obtain two images of the object to be measured from different positions, and calculates the positional deviation between the corresponding points of the images based on the principle of parallax. A method for obtaining 3D geometric information of an object. Binocular stereo vision cameras have low hardware requirements, ordinary RGB cameras can be used, and are suitable for indoor and outdoor, as long as the light is suitable. However, its shortcomings are also very obvious, such as being very sensitive to ambient lighting, not suitable for scenes with monotonous lack of texture, high computational complexity, and the baseline limits the measurement range, and so on.

Smartphone 1600 may also include other sensors 1607, such as light sensors, gravity sensors, proximity sensors, fingerprint sensors, and the like. In addition, the smart phone 1600 is usually equipped with a gyro sensor, an acceleration sensor, etc. in order to detect its motion state or attitude.

It should be understood that the above merely schematically describes a smartphone as an example of a handheld device and its representative components, and does not mean that these components are all necessary for a handheld device according to the present disclosure.

According to embodiments of the present disclosure, the handheld device can adaptively present the UI according to which hand the user is operating the device with. An electronic device for adaptive UI presentation and a method for performing the same according to the present disclosure will be described below in conjunction with FIGS. 3A and 3B .

FIG. 3A shows a block diagram of electronic device 1000 . Electronic device 1000 includes processing circuit 1001 and potentially other circuits. The processing circuit 1001 may, for example, be implemented as a processor such as the processor 1601 described above. As shown in FIG. 3A, the processing circuit 1001 includes an acquisition unit 1002, a determination unit 1003, and a presentation unit 1004, and may be configured to perform the method shown in FIG. 3B.

The acquisition unit 1002 of the processing circuit 1001 is configured to acquire distance data associated with one or more physical characteristics of the user of the handheld device (ie, perform step S1001 in FIG. 3B ).

According to embodiments of the present disclosure, the physical features of the user may include various facial features or other physical features. In one example, the physical features of the user include features such as forehead, nose tip, mouth, chin, and the like. In another example, the user's physical features include a pair of left-right symmetrical features, such as eyes, shoulders, and the like. For which body features the acquiring unit 1002 acquires the distance data may depend on the specific determination method, which will be described in detail below.

The distance data associated with the physical features of the user may be obtained from depth images of the user captured by the depth camera. The depth image takes the distance (depth) from the light receiver of the depth camera (eg TOF sensor) to the user's body as the pixel value. That is, the depth image contains the distance information of the captured user's body features. Typically, a depth image can be represented as a two-dimensional point cloud, when the two directions of the sensor plane of the depth camera are considered as X, Y directions (eg, the X direction for the horizontal direction and the Y direction for the vertical direction) and the sensor plane is regarded as the X and Y directions. When the normal direction (depth direction) of the plane is regarded as the Z direction to establish a coordinate system, the pixel value of the depth image can be expressed as (x _i , y _i , z _i ), where x _i , y _i , z _i are three-dimensional distance in the direction.

In one example, the processing circuit 1001 may perform image recognition processing on the depth image obtained by the depth camera to recognize the required body feature. There are many such image recognition processes in the prior art, such as various classification methods, artificial intelligence, etc., which will not be repeated here. Then, the obtaining unit 1002 may obtain corresponding pixel values from the depth image as distance data associated with the body feature.

In another example, the handheld device may utilize an RGB camera and a depth camera to simultaneously obtain an image of the user, and the RGB image obtained by the RGB camera and the depth image obtained by the depth camera may be aligned with each other. There are even cases where an RGB camera and a depth camera are integrated, such a camera can obtain both the user's RGB value and the distance as a pixel value. The processing circuit 1001 can perform image recognition processing on the RGB image to recognize the required body features. Then, based on the correspondence between the pixel points of the depth image and the RGB image, the obtaining unit 1002 may obtain the corresponding pixel value from the depth image as the distance data associated with the body feature.

The determination unit 1003 may be configured to determine the hand of the user currently operating the handheld device based on the distance data acquired by the acquisition unit 1002 (ie, perform step S1002 in FIG. 3B ). Specifically, based on distance data associated with one or more physical characteristics of the user, the determination unit 1003 may first determine the orientation of the handheld device relative to the user. Several examples of determining the orientation of a handheld device relative to a user are presented here.

In one example, the determination unit 1003 may calculate the azimuth of the body feature relative to the handheld device based on the distance data associated with the body feature. 4A and 4B show front and top views, respectively, of the orientation of the nose tip relative to the handheld device when the user holds the handheld device with the right and left hands. It should be understood that although FIGS. 4A and 4B use the tip of the nose as an example of a body feature, the present disclosure is not limited thereto, and the body feature may also be a forehead, a mouth, a chin, and the like.

As shown in the figure, if a coordinate system is established using the depth camera's sensor as the center, the azimuth of the user's nose tip can be calculated as

Where x is the distance of the nose tip in the X direction (eg horizontal direction), z is the distance of the nose tip in the Z direction (eg the normal direction of the sensor plane, that is, the depth direction). As shown in Figure 4A, when the handheld device is located to the right of the user, since x is a positive value, θ should be greater than 0; while as shown in Figure 4B, when the handheld device is located to the left of the user, due to If x is negative, θ should be less than 0. Thus, by comparing the calculated azimuth to a predetermined threshold (eg, 0), the orientation of the handheld device relative to the user can be determined.

It should be understood that the predetermined threshold here is not limited to 0, but may be appropriately set in consideration of tolerances, for example, if the calculated azimuth angle θ is greater than 5°, 10° or other thresholds, it is determined that the handheld device is located at the user , and if the calculated azimuth angle θ is less than -5°, -10°, or other thresholds, it is determined that the handheld device is located to the right of the user.

In another example, the judging unit 1003 may consider more than one body feature, but may consider two or more body features, such as eyes, shoulders, and the like. 5A and 5B illustrate an example of calculating the azimuth angle for the eye. As shown in Figure 5A, when the handheld device is located to the user's right, the azimuth angle θR of the right eye should be smaller than the azimuth angle θL of the left eye; and when the handheld device is located to the user's left, the azimuth angle of the right eye θR should be greater than the azimuth angle θL of the left eye. Therefore, by comparing the azimuth angles of the two eyes, it can be determined whether the handheld device is located to the left or right of the user.

In fact, as can be seen from Figures 5A and 5B, the direction in which the handheld device is located relative to the user always corresponds to a body feature (eg eyes) having a smaller azimuth. For example, in FIG. 5A, the handheld device is located to the right of the user, which corresponds to the right eye with a smaller azimuth, while in FIG. 5B, the handheld device is located to the left of the user, which corresponds to a lower azimuth Small azimuth left eye. This means that using the hand has an ipsilateral relationship to body features with smaller azimuths.

In yet another example, the determination unit 1004 may calculate the distance between the body feature and the handheld device based on distance data associated with a pair of body features. 6A and 6B illustrate an example of calculating distances for eyes. It should be understood that the physical features considered here may not be limited to the eyes, but may be a pair of any physical features that are symmetrical from left to right, such as shoulders. By establishing the coordinate system as above, the distance of either eye can be calculated as

Where x is the distance of the eye in the X direction (eg horizontal direction), z is the distance of the eye in the Z direction (eg the normal direction of the sensor plane, that is, the depth direction).

As shown in Figure 6A, when the handheld device is located to the right of the user, the distance dR for the right eye should be smaller than the distance dL for the left eye, and when the handheld device is located to the left of the user, the distance dR for the right eye should be greater than the left eye distance dR Eye distance dL. Therefore, by comparing the distance between the two eyes, it can be determined whether the handheld device is located to the left or right of the user. Likewise, it can be found that the direction in which the handheld device is located relative to the user always corresponds to the eye with a smaller distance. This means that using the hand has an ipsilateral relationship to the body feature with a smaller distance.

Although several exemplary methods of determining the orientation of a handheld device relative to a user based on depth data associated with body features are described above, the present disclosure is not so limited. Any method can be used as long as it achieves the same purpose. For example, a machine learning approach can be employed, where a model, such as a neural network model, can be constructed using depth images or distance data as an input training set and the orientation of the handheld device relative to the user as an output training set and, in use, judge The unit 1003 may input the corresponding depth image or the distance data obtained by the obtaining unit 1002 to the model to obtain the prediction output of the model.

After determining the direction of the handheld device relative to the user, the determination unit 1003 can determine which hand the user is using to hold the handheld device. Typically, when the user holds the handheld device with the left hand, the device is located in front of the user to the left, and when the user holds the handheld device in the right hand, the device is located in the front right of the user. Thus, the determination unit 1003 can determine the hand of the user to operate the handheld device according to the direction of the handheld device relative to the user.

There may be situations where the user holds the handheld device with the left hand but the handheld device is located to the user's right or vice versa. In this case, the judgment unit 1003 may obtain an erroneous judgment result. According to an embodiment of the present disclosure, the processing circuit 1001 may also correct the judgment result of the judgment unit 1003 using sensing data from additional sensors other than the depth camera.

In one example, the processing circuit 1001 can use the sensing data of the acceleration sensor to detect the motion trajectory of the handheld device. If the handheld device moves from left to right, crosses the front of the user and is located to the right of the user, then The processing circuit 1001 can correct the judgment result of the judgment unit 1003 to use the left hand in combination with the detected motion trajectory, and vice versa.

In another example, the processing circuit 1001 can use the sensing data of the gyro sensor to detect the posture of the handheld device, and integrate ergonomics and face position to comprehensively judge the user's hand.

In yet another example, the processing circuit 1001 may control to display the judgment result of the judgment unit 1003, and prompt the user to give confirmation. For example, "Detected that the current using hand is a left hand?" may be displayed on the screen, and the user may confirm by clicking "Yes" or "No".

Returning to FIGS. 3A and 3B , according to the determined hand, the presentation unit 1004 of the processing circuit 1001 may present a UI suitable for the user's hand operation (ie, perform step S1003 in FIG. 3B ). For example, when it is determined that the user is holding the handheld device with the left hand, the presenting unit 1004 may present the UI components requiring user operation as being close to the left side of the screen so as to be within reach of the fingers of the user's left hand. On the contrary, when it is judged that the user is holding the handheld device with the right hand, the presenting unit 1004 may present the UI components requiring user operation as being close to the right side of the screen so as to be within reach of the fingers of the user's right hand.

The hand-use judgment method of the present disclosure can only use the depth camera commonly equipped on the device to infer which hand the user is holding and operating the handheld device with, without the need to equip a specific sensor. In addition, the hand judgment method of the present disclosure is less affected by lighting conditions, and can ensure a wide range of application scenarios and high accuracy.

The method process described above can be performed in many occasions to provide convenience for the user to operate the UI.

For example, when the user unlocks the handheld device, the technology of the present disclosure can be performed, for example, when the user performs facial recognition or presses the unlock key, the depth camera is activated to capture a depth image, and the hand is determined based on the captured depth image, so as to UI suitable for user operation is directly presented on the unlocked screen.

Or, when a change in the posture or usage scenario of the handheld device is detected through a gyro sensor or an acceleration sensor, for example, the user changes from standing to lying down, the user picks up the phone, the user's left and right hands are swapped, etc. Execution of the method process of the present disclosure is triggered.

Additionally, use of the present disclosure may be initiated when a reversal change in the orientation of the handheld device relative to the user is detected by a camera or other sensor, eg, moving from the user's left to right or right to left Hand judgment, the handheld device can present a query to the user whether to change the hand on the screen, if the response received from the user is to confirm the change of the hand, then present the changed UI, otherwise, if the user responds that the hand has not changed, the UI No need to change either.

According to the embodiments of the present disclosure, various implementations of implementing the concepts of the present disclosure can be conceived, including but not limited to:

1) An electronic device for a handheld device, comprising: a processing circuit configured to: obtain distance data associated with one or more physical characteristics of a user of the handheld device; based on the distance data determining the direction of the handheld device relative to the user; based on the determined direction, judging the hand of the user currently operating the handheld device; and presenting a hand suitable for the user according to the judgment result The user interface (UI) of the operation.

2) The electronic device according to 1), wherein the processing circuit is further configured to: identify the one or more body features based on a depth image about the user obtained by a depth camera, and obtain the distance data.

3) The electronic device according to 1), wherein the processing circuit is further configured to: identify the one or more body features based on an RGB image of the user obtained by an RGB camera; and Distance data associated with the identified one or more body features is obtained from the depth image obtained by the depth camera.

4) The electronic device according to 1), wherein the processing circuit is further configured to: based on the distance data, calculate an azimuth angle of one or more body features of the user relative to the handheld device , and determine the orientation of the handheld device relative to the user based on the azimuth.

5) The electronic device according to 1), wherein the processing circuit is further configured to: based on the distance data, calculate and compare a pair of left-right symmetrical body features of the user with the hand-held device. and determine the orientation of the handheld device relative to the user based on the distance.

6) The electronic device according to 1), wherein the physical features include at least one of the following: nose tip, chin, forehead, mouth, eyes, shoulders.

7) The electronic device according to 1), wherein the processing circuit is further configured to: when the direction of the handheld device relative to the user is reversed and changed, present to the user whether to change the usage A hand query; receiving a response to the query from the user; determining whether to change the user interface (UI) according to the received response.

8) The electronic device according to 1), wherein the depth camera is one of the following: a time-of-flight (TOF) camera, a structured light camera, and a binocular stereo vision camera.

9) A handheld device, comprising: a depth camera for obtaining a depth image about a user of the handheld device; and a processing circuit configured to: based on the depth image, obtain a depth image related to the user or distance data associated with multiple body features; determine the direction of the handheld device relative to the user based on the distance data; determine the hand of the user currently operating the handheld device based on the determined direction ; present a user interface (UI) suitable for the user's hand-operated operation according to the judgment result.

10) The handheld device according to 9), wherein the processing circuit is further configured to: based on the depth image, identify the one or more body features, and obtain the distance data.

11) The handheld device according to 9), further comprising an RGB camera for capturing an RGB image, wherein the processing circuit is further configured to: based on the RGB image about the user captured by the RGB camera, identify the one or more body features; and obtaining distance data associated with the identified one or more body features from a depth image obtained by the depth camera.

12) The handheld device according to 9), wherein the processing circuit is further configured to: based on the distance data, calculate an orientation of one or more physical features of the user relative to the handheld device angle and determine the orientation of the handheld device relative to the user based on the azimuth angle.

13) The handheld device according to 9), wherein the processing circuit is further configured to: based on the distance data, calculate and compare a pair of left-right symmetrical body features of the user with the handheld device distance between and determine the orientation of the handheld device relative to the user based on the distance.

14) The handheld device according to 9), wherein the depth camera is one of the following: a time-of-flight (TOF) camera, a structured light camera, and a binocular stereo camera.

15) A method for a handheld device, comprising: obtaining distance data associated with one or more physical characteristics of a user of the handheld device; the direction of the user; based on the determined direction, judging the hand of the user currently operating the handheld device; and presenting a user interface (UI) suitable for the hand operation of the user according to the judgment result.

16) A non-transitory computer-readable storage medium storing executable instructions which, when executed, implement the method according to 15).

Exemplary embodiments of the present disclosure have been described above with reference to the accompanying drawings, but the present disclosure is not limited to the above examples, of course. Those skilled in the art may find various changes and modifications within the scope of the appended claims, and it should be understood that they will naturally come under the technical scope of the present disclosure.

For example, a plurality of functions included in one unit in the above embodiments may be implemented by separate devices. Alternatively, multiple functions implemented by multiple units in the above embodiments may be implemented by separate devices, respectively. Additionally, one of the above functions may be implemented by multiple units. Needless to say, such a configuration is included in the technical scope of the present disclosure.

In this specification, the steps described in the flowcharts include not only processing performed in time series in the stated order, but also processing performed in parallel or individually rather than necessarily in time series. Furthermore, even in the steps processed in time series, needless to say, the order can be appropriately changed.

Although the present disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims. Furthermore, the terms "comprising", "comprising" or any other variation thereof in embodiments of the present disclosure are intended to encompass a non-exclusive inclusion such that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also Include other elements not expressly listed, or which are inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Claims

An electronic device for a handheld device, comprising:

processing circuitry, configured as:

obtaining distance data associated with one or more physical characteristics of the user of the handheld device;

determining an orientation of the handheld device relative to the user based on the distance data;

Based on the determined direction, determining the hand of the user currently operating the handheld device; and

According to the result of the judgment, a user interface (UI) suitable for the user's manual operation is presented.
The electronic device of claim 1, wherein the processing circuit is further configured to:

The one or more body features are identified based on a depth image of the user obtained by a depth camera, and the distance data is obtained.
The electronic device of claim 1, wherein the processing circuit is further configured to:

identifying the one or more physical features based on an RGB image of the user obtained by an RGB camera; and

Distance data associated with the identified one or more body features is obtained from the depth image obtained by the depth camera.
The electronic device of claim 1, wherein the processing circuit is further configured to:

Based on the distance data, an azimuth angle of one or more physical features of the user relative to the handheld device is calculated, and an orientation of the handheld device relative to the user is determined based on the azimuth angle.
The electronic device of claim 1, wherein the processing circuit is further configured to:

Based on the distance data, a distance between a pair of left-right symmetrical body features of the user and the handheld device is calculated and compared, and an orientation of the handheld device relative to the user is determined based on the distance.
The electronic device of claim 1, wherein the physical feature comprises at least one of the following: nose tip, chin, forehead, mouth, eyes, shoulders.
The electronic device of claim 1, wherein the processing circuit is further configured to:

presenting a query to the user whether to change the hand used when the orientation of the handheld device relative to the user is reversed;

receive a response to a query from the user;

Whether to change the user interface (UI) is determined according to the received response.
The electronic device of claim 1, wherein the depth camera is one of the following: a time-of-flight (TOF) camera, a structured light camera, and a binocular stereo camera.
A handheld device comprising:

a depth camera for obtaining depth images about a user of the handheld device; and

processing circuitry, configured as:

obtaining distance data associated with one or more physical features of the user based on the depth image;

determining an orientation of the handheld device relative to the user based on the distance data;

Based on the determined direction, determine the hand of the user currently operating the handheld device;

According to the result of the judgment, a user interface (UI) suitable for the user's manual operation is presented.
The handheld device of claim 9, wherein the processing circuit is further configured to:

Based on the depth image, the one or more body features are identified, and the distance data is obtained.
The handheld device of claim 9, further comprising an RGB camera for capturing RGB images,

Wherein, the processing circuit is further configured to:

identifying the one or more physical features based on an RGB image of the user captured by an RGB camera; and

Distance data associated with the identified one or more body features is obtained from the depth image obtained by the depth camera.
The handheld device of claim 9, wherein the processing circuit is further configured to:

Based on the distance data, an azimuth angle of one or more physical features of the user relative to the handheld device is calculated, and an orientation of the handheld device relative to the user is determined based on the azimuth angle.
The handheld device of claim 9, wherein the processing circuit is further configured to:

Based on the distance data, a distance between a pair of left-right symmetrical body features of the user and the handheld device is calculated and compared, and an orientation of the handheld device relative to the user is determined based on the distance.
The handheld device of claim 9, wherein the depth camera is one of: a time-of-flight (TOF) camera, a structured light camera, a binocular stereo camera.
A method for a handheld device, comprising:

obtaining distance data associated with one or more physical characteristics of the user of the handheld device;

determining an orientation of the handheld device relative to the user based on the distance data;

based on the determined direction, determining the hand of the user currently operating the handheld device; and

According to the result of the judgment, a user interface (UI) suitable for the user's manual operation is presented.
A non-transitory computer-readable storage medium storing executable instructions that, when executed, implement the method of claim 15 .