TW201931096A - Interaction method for user interface and electronic device thereof - Google Patents

Abstract

An interaction method for a user interface of an electronic device, the user interface including a display device, includes detecting a biometric signal of head of a user and controlling the display device to display a page according to the biometric signal of head; detecting a biometric signal of eyes of the user and controlling a position of an indicator on the page according to the biometric signal of eyes; and detecting an instruction of a user and controlling the electronic device to perform a specific function according to the position of the indicator on the page when the instruction is detected.

Description

Interactive method and electronic device for user interface

The present invention is directed to an interactive method for a user interface, and more particularly to a method for allowing a user to interact with a user interface without the need for hand manipulation.

The user interface of the electronic device often includes multiple hierarchies. When the user operates, it may switch between different pages in the same layer or switch to the next page or screen. For example, in a menu mode, a user may click on a particular item to perform a particular function, or click on an item to proceed to the next level of the menu. Generally speaking, when operating on a computer or a mobile phone, the user needs to perform a touch gesture or input an instruction through an input interface such as a keyboard or a mouse. In other words, the traditional mobile phone or computer operation mode needs to be performed with one hand or both hands, and the user needs to watch the message on the screen for a long time to realize the interaction with the user interface of the electronic device. However, the above operation mode cannot be applied to driving, and if the driver needs to vacate one or both hands to operate the electronic device, it is easy to be distracted and affect driving safety. In view of this, it is necessary to propose a more suitable electronic device interaction method for use by motorists.

The main object of the present invention is to provide a method for allowing a user to interact with a user interface without using a hand operation to solve the problem of easy distraction of the driver and reduce the complexity of the driver operating the electronic device. And improve driving safety.

The present invention discloses an interactive method for a user interface of an electronic device. The user interface includes a display device. The interaction method includes: detecting a biometric signal of a user's head, and according to the head The biometric signal controls the display device to display a page; detects an eye biometric signal of the user, and controls an indicator on the page according to the eye biometric signal; and detects one of the users And, when the instruction is detected, controlling the electronic device to perform a specific function according to the position of the indicator on the page.

The invention further discloses an electronic device comprising a user interface and a processing device. The user interface is for interacting with a user having a display device and a sensing device. The sensing device is configured to detect a biometric signal of the user's head and an eye biometric signal. The processing device is coupled to the user interface for performing the following steps: controlling the display device to display a page according to the biometric identification signal of the head; and controlling an indicator on the page according to the biometric identification signal of the eye a location; and upon detecting an instruction of the user, controlling the electronic device to perform a specific function based on the location of the indicator on the page.

The present invention further discloses an interactive method for a user interface of an electronic device, the user interface comprising a display device, the interaction method comprising: detecting an eye biometric signal of a user, and according to the eye The biometric identification signal controls a position of an indicator on a page displayed by the display device; and detects a facial command of the user, and when the facial command is detected, according to the indicator on the page The upper position controls the electronic device to perform a specific function; wherein the facial command is composed of a plurality of facial actions.

The invention further discloses an electronic device comprising a user interface and a processing device. The user interface is for interacting with a user having a display device and a sensing device. The sensing device is configured to detect an eye biometric signal and a facial command of the user, wherein the facial command is composed of a plurality of facial actions. The processing device is coupled to the user interface for performing the following steps: controlling, according to the eye biometric signal, a position of an indicator on a page displayed by the display device; and detecting the facial command The electronic device is controlled to perform a specific function based on the location of the indicator on the page.

Figure 1 is an embodiment of the invention. The electronic device 10 includes a user interface 100 and a processing device 110. The user interface 100 provides a medium for interacting with the user, and includes a display device 102 and a sensing device 104. The electronic device 10 can also optionally include a user input device 120. The sensing device 104 can be used to detect biometric signals of the user's head and eyes. The processing device 110 is coupled to the user interface 100, and controls the display device 102 to display a page according to the head biometric signal; and controls the position of an indicator on the page according to the eye biometric signal. The user input device 120 can receive a user's command to control the electronic device 10 to perform a particular function based on the location of the indicator on the page.

In one embodiment, the sensing device 104 can be used to detect the user's eye biometric signal and detect the user's facial biometric signal as a facial command. The processing device 110 can control the position of an indicator on a page displayed by the display device 102 according to the eye biometric signal, and control the execution of the electronic device 10 according to the position of the indicator on the page when the face command is detected. A specific function. The above facial command can be composed of a plurality of facial actions.

In one embodiment, the sensing device 104 can be used to detect biometric signals of the user's head, eyes, and face. The processing device 110 is coupled to the user interface 100, and can control the display device 102 to display a page according to the head biometric signal; control the position of an indicator on the page according to the eye biometric signal; and according to the facial biometric signal And the location of the indicator on the page, the control electronics 10 performs a particular function.

In one embodiment, the user's head biometric signal detected by the sensing device 104 includes a head gesture, a head position, a head movement, and any combination of head posture, position, and/or movement. The head position represents the relative position between the head and the sensor or represents the relative position between the head and the display. The head posture represents a nod, a head or a side head.

In one embodiment, the user's eye biometric signal detected by the sensing device 104 includes a gaze vector and a gaze point. The gaze vector represents the direction of the eye gaze derived by head position, head posture, iris reflected light, iris position, and/or pupil position. The gaze vector can be implemented in a world coordinate system, a facial coordinate system, or a screen coordinate system. The gaze point represents the intersection of the gaze vector and the screen.

In one embodiment, the user's facial biometric signal detected by the sensing device 104 includes a facial feature, a facial motion, a series of facial motions, or a combination of any facial features and/or motions.

In an embodiment, the sensing device 104 can include any type of sensor for obtaining various biometric signals, such as a visible light camera, an infrared camera, a photo sensor, or a combination of sensors, without limitation. this.

In one embodiment, the processing device 110 can convert the biometric signals of the user's head, eyes, and/or face into different commands to control the display displayed on the display device 102.

In an embodiment, the manner of displaying the page includes controlling a pan of the display by a specific head motion. Specific head motions may be achieved by a combination of head posture, head position, head movement, or any head posture, position, and/or movement. Taking FIG. 2A as an example, when the user's head is tilted to the left, the displayed screen can be controlled to shift to the left.

In an embodiment, the manner of displaying the page includes controlling zoom-out or zoom-in of the display by a specific head motion. Specific head motions may be achieved by a combination of head posture, head position, head movement, or any head posture, position, and/or movement. Taking FIG. 2B as an example, when the user's head moves toward the screen, the screen for controlling the display can be enlarged.

In order to implement the panning and zooming in the above embodiment, the background area of the active area on the screen may present a virtual image, the active area is a subset of the virtual image, and the active area has a smaller area range than the virtual image.

In an embodiment, the step of controlling the position of an indicator on the page can be implemented in a variety of ways. For example, a cursor may be used on the active area of the screen, or a specific area, window, and/or image may be indicated by recognizable colors, borders, convexities, or concaves. Taking the 2C figure as an example, on the screen 20, the cursor and the frame line are used to indicate a specific item.

In an embodiment, the user input device 120 can receive a user's command in any manner, such as by pressing a physical button, inputting a specific gesture, or inputting through a touch screen, etc., but is not limited thereto.

In an embodiment, the step of controlling the electronic device 10 to perform a specific function according to the position of the indicator on the page may be implemented in various manners. For example, the region, window, and/or image selected based on the indicator may indicate that a particular function is being performed, the region, window, and/or image may correspond to a particular function or instruction. For example, the calling function can be to open the application, return to the main menu, pull down the menu, restart, etc., but is not limited thereto.

In this case, the user can operate the electronic device 10 by using the biometric signals of the head, the eyes, and/or the face, and simultaneously view the content of the screen on the display device 102 to achieve the user and user interface 100. interactive.

In an embodiment, the display device 102 can be any type of display, such as a liquid crystal display (LCD), an Organic Light Emitting Diode Display (OLED Display), or the like.

In an embodiment, the processing device 110 can be a central processing unit (CPU), a microprocessor, a micro controller unit (MCU), or a specific application integrated circuit. (Application-Specific Integrated Circuit, ASIC), which is not limited to being implemented by hardware or software.

In an embodiment, the electronic device 10 can be a vehicle electronic device.

FIG. 3 is a schematic diagram of an interaction process 30 according to an embodiment of the present invention. The interactive process 30 can be used in an electronic device, such as the electronic device 10 in Figure 1, for interacting with a user. As shown in Figure 3, the interactive process 30 includes the following steps:

Step 300: Start.

Step 302: Detecting a head biometric signal of the user, such as a head posture, a head position, a head movement, or a combination of any head posture, position, and/or movement, and according to the detected head The biometric signal control display device displays a page.

Step 304: detecting one of the user's eye biometric signals, such as a gaze vector, a gaze point, or a combination of a gaze vector and a gaze point, and controlling an indicator on the page according to the detected eye biometric signal. position.

Step 306: Detect one of the user's instructions or the facial biometric signal, and when detecting the command or the facial biometric signal, control the electronic device to perform a specific function according to the position of the indicator on the page.

Step 308: End.

According to the interaction process 30, the sensing device 104 can detect the user's head biometric signal, and the processing device 110 can control the display device 102 to display a page according to the head biometric signal. For example, the processing device 110 can control the screen displayed on the display device 102 to scroll or move in a specific direction according to the head biometric signal. For example, the driver may be looking for an application, contact, or mode switching option on a menu. If the sensing device 104 detects that the driver's head is moving to the left, the screen on the display device 102 can be moved to the left. To display the more right option; conversely, if the sensing device 104 detects that the driver's head is moving to the right, the screen on the display device 102 can be moved to the right to display the more left option. In another embodiment, the display device 102 can also be controlled to display the option on the left side by the left head motion, and the display device 102 can be controlled to display the right option by the right head motion. Alternatively, the driver may be browsing a certain page, and the processing device 110 may scroll the page according to the detected head biometric signal, just as the user scrolls the page with the mouse wheel, but in this case, You don't need to do it yourself.

In another embodiment, the processing device 110 can also control zoom-in or zoom-out displayed on the display device 102 according to the head biometric signal. For example, when it is detected that the driver's head is close to the sensing device 104, the screen can be enlarged to highlight the option that the user wants to view; when the driver's head is detected to be away from the sensing device 104, the control screen is zoomed out. The screen contains more options for the driver to choose. In addition, in addition to detecting the direction in which the head moves, the processing device 110 can also detect whether the head is tilted or turned to the left or right to determine the head motion. For example, the left head tilt can control the page to scroll to the left, and the right tilt can control the page to scroll to the right. Any manner in which the display of the page can be controlled by the head biometric signal is within the scope of the present invention.

4A-4C illustrate a detailed operation mode of controlling the display device 102 to display a page 400 by the head biometric signal. The page 400 contains a form containing a number of options, such as settings, information mode, GPS mode, and the like. As shown in FIG. 4A, page 400 only displays the complete "Information Mode" option, and other options are not fully displayed on page 400. When the user wants to view other options, the content displayed by page 400 can be controlled by head gestures, positions, and/or movements. As shown in Fig. 4B, the user can move the head to the right. When the sensing device 104 detects that the head is shifted to the right, the content of the screen can be tilted to highlight the option on the right side, and the user can easily select the "GPS mode" option on the right side. As shown in FIG. 4C, the user can move the head to the left. When the sensing device 104 detects that the head is shifted to the left, the content of the screen can be tilted to the other direction to highlight the option on the left side. The user can easily select the "Settings" option on the left.

In the above embodiment, the form displayed by the page 400 is arranged in a row for a plurality of options; and in another embodiment, the plurality of options may be arranged in a circular arc shape. In this case, when the user's head is detected to move to the right, the controllable option moves in a clockwise direction; when the user's head is detected to move to the left, the controllable option moves in a counterclockwise direction.

In this example, the electronic device 10 can be, for example, a vehicle electronic device that can be easily manipulated by a vehicle driver. Thus, display device 102 can be a dashboard of a vehicle that displays pages 400 and forms containing various information related to driving. For example, after entering the "information mode" option, the display device 102 can display various driving information, such as weather, time, driving distance, etc.; after entering the "GPS mode" option, the display device 102 can display GPS navigation information, such as a map and Destination information, etc.; the "Settings" option is used to enter the settings menu to make various settings. The options on the form are not limited to the description of the above embodiments, and in other embodiments, options such as a speedometer, a driving recorder, a linking mobile device, and the like may also be included. All of the above options can be set in the form of Figures 4A-4C to control the form to scroll or move left or right by head gesture, position and/or movement to display different options.

The sensing device 104 can detect the eye biometric signal of the user, and the processing device 110 can control the position of an indicator on the page according to the eye biometric signal. The indicator can be presented in various forms, such as a cursor, a pattern, a frame line, or a particular color to indicate the item or area that the user desires to select. Figure 5 illustrates that when the user gaze at the "Information Mode" option, the processing device 110 can control the indicator to move to the "Information Mode" option. In this example, the indicator is displayed in a bolder frame on the indicated option, but those of ordinary skill in the art should understand that the indicator can also be presented in other ways, for example by comparison. A bright border is used to mark the option you want to select, or to mark it in a specific color on the option you want to select, as long as the indicator is displayed in a way that lets the user know exactly where the indicator is. In another embodiment, the indicator can also be presented in the form of a cursor and the cursor can be moved in any direction depending on where the user is staring. The eye movement is controlled by eye gaze as if the user is moving with the mouse or sliding touch gesture control indicator.

The user input device 120 is operable for the user to input an instruction to cause the processing device 110 to control the electronic device 10 to perform a particular function based on the position of the indicator on the page. In an embodiment, the instruction to control the electronic device 10 to perform a specific function is a facial command of the user detected by the sensing device 104, and the manner of controlling the movement of the indicator by eye gaze can be analogized to pass the mouse. Or sliding the touch gesture control indicator to move, then controlling the electronic device 10 to perform a specific function by using a face command can be analogized to a click mouse, a keyboard or a touch screen confirmation key to perform a specific function.

Face commands can be implemented in a variety of ways. For example, the facial command can be any facial expression, such as a smile, a laugh, a mouth, a side of the nose, etc., and a facial expression can also be a dynamic combination of a series of facial actions. The facial expression can be preset. For example, the user can set a smile (ie, the mouth is raised) as a facial command. When the sensing device 104 detects that the user's mouth is raised, the facial command can be judged to receive a smile. . Taking the pictures 4A to 5 as an example, the following is an example. The user can move between different options through the eye control indicator. If the user wants to select the "Information Mode" option, the control indicator can be moved to the "Information Mode" option and start to smile. After the sensing device 104 detects the smile, the processing device 110 can determine that a confirmation command is received, thereby controlling the electronic device 10 to enter the information mode, for example, displaying various driving information through the display device 102.

In another embodiment, each user can set a respective facial expression. In addition to determining facial expressions, the processing device 110 must determine personal facial features. In one embodiment, a car may be used by both spouses, and the couple may use different facial expression settings, such as the husband setting a smile as a confirmed facial command, and the wife setting a wink as a confirmed facial command. When the husband drives the car, the system can judge the driver as the husband according to the facial features, and judges that the confirmation command is received when the smiling expression is detected; when the wife drives the car, the system can judge the driver as the wife according to the facial features, and It is judged that a confirmation command is received when a blinking expression is detected.

The above detection of the head, eyes and face can be achieved by Computer Vision technology. The sensing device 104 can obtain an image including the user's head. Then, the processing device 110 can downsample the image to reduce the amount of data, and then take the user's head image through the face detection technology. To set a Bounding Box corresponding to the head image. Downsampling can reduce the amount of data and increase the speed of subsequent processing. However, in another embodiment, the step of downsampling can be omitted, and the face detection technology can be directly performed on the image obtained by the sensing device 104. The processing device 110 intercepts the image content inside the bounding box for subsequent processing. The judgments related to the head, eye and face actions can be obtained from the internal image of the bounding box.

The processing device 110 can obtain the feature point positions of various face organs, such as eyes, mouth, nose, etc., through the Facial Landmark Detection technology. The processing device 110 can obtain the head biometric signal of the user to determine the head motion, such as the position of the head, whether it is moving, rotating or tilting. The processing device 110 can determine the position of the pupil and/or the light reflected by the pupil for the image of the eye to obtain the gaze vector and the gaze point. The processing device 110 may further determine the facial expression according to a local position change of the face organ or according to a series of dynamic changes of the facial features, for example, when it is detected that the corner of the mouth is raised, it is determined to be a smile, and according to the whole body of each face. Position to determine facial features. The invention can control the operation of the electronic device by detecting the biometric signals of the head, the eyes and the face, and the detection of the head, the eyes and the face can adopt various feasible methods, which should not be Limitations of the invention.

The present invention provides a method for allowing a user to interact with a user interface without the use of a hand operation, and those skilled in the art can modify or vary it without limitation. For example, in the above embodiment, the system detects the user's head, eyes, and face to obtain an instruction to display a corresponding page on the user interface while performing a specific function on the electronic device. The user can interact with the electronic device without inputting a gesture by gesture. In another embodiment, the user can also perform an acknowledgment command by pressing a physical button or a specific gesture, or a dictation command to control the electronic device to perform a specific function (pressing a physical button, a gesture, and a dictation command can be used instead of the foregoing. Face command). In general, the vehicle electronic system can set a plurality of physical buttons on the steering wheel, and the physical button can be used as a medium for input confirmation commands without reducing the convenience and driving safety, and the driver can easily press the steering wheel. Press the button on to enter. Alternatively, the processing device 110 may also have a voice detection/recognition function for the user to input voice commands. In addition, the interactive method proposed by the present invention can be applied not only to a vehicle electronic system but also to other types of electronic systems, such as game machines, e-book readers, and the like. It is within the scope of the present invention to allow the user interface to interact with the user by detecting the user's head, eye and face movements and/or features.

In an embodiment of the present invention, the user interface of the electronic device can display the page by detecting the biometric signal of the user's head, and control the scrolling, moving, and/or zooming of the page as the head moves. . The user's eye biometric signal is then detected to control the position of an indicator on the page based on the location of the user's eye gaze. Then, when a particular facial command is detected, the electronic device can perform a particular function, such as entering a menu or mode of operation. In this case, the user can input commands through the head, eyes and face without having to operate by hand to achieve interaction with the user interface. The above are only the preferred embodiments of the present invention, and all changes and modifications made to the scope of the present invention should be within the scope of the present invention.

10‧‧‧Electronic devices

100‧‧‧User interface

102‧‧‧ display device

104‧‧‧Sensing device

110‧‧‧Processing device

120‧‧‧User input device

30‧‧‧Interactive process

300 to 308 ‧ steps

400‧‧‧ page

FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. FIG. 2A is a schematic diagram of controlling display screen shift by head motion according to an embodiment of the present invention. FIG. 2B is a schematic diagram of controlling the reduction or enlargement of the display screen by the head motion according to the embodiment of the present invention. FIG. 2C is a schematic diagram of an indicator according to an embodiment of the present invention. FIG. 3 is a schematic diagram of an interaction process according to an embodiment of the present invention. 4A-4C illustrate a detailed operation mode of controlling a display device to display a page by a head motion. Figure 5 is a schematic diagram of the user controlling the movement of the indicator by gazing.

Claims (16)

An interactive method for a user interface of an electronic device, the user interface includes a display device, the interaction method includes: detecting a biometric signal of a head of a user, and identifying the biometric according to the head The signal controls the display device to display a page; detecting an eye biometric signal of the user, and controlling the position of the indicator on the page according to the eye biometric signal; and detecting an instruction of the user And when detecting the instruction, controlling the electronic device to perform a specific function according to the position of the indicator on the page. The interactive method of claim 1, wherein the step of controlling the display device to display the page according to the biometric identification signal comprises: controlling, according to the biometric identification signal of the head, a screen displayed on the display device to a specific direction Scroll or move, or control the screen displayed on the display device to zoom in or out to display the page. The interaction method of claim 1, wherein the indicator comprises a cursor, a pattern, a frame line, or a specific color for indicating an item or region to be selected. The interactive method of claim 1, wherein the instruction comprises a static facial command or a dynamic combination of a plurality of facial actions. The interactive method of claim 1, wherein the form displays a form, the form includes a plurality of options, and when the instruction is detected, controlling the electronic device to execute according to the position of the indicator on the page. The step of the specific function includes: when detecting the instruction, selecting one of the plurality of options indicated by the indicator according to the position of the indicator on the page. An electronic device includes: a user interface for interacting with a user, the user bread comprises: a display device; and a sensing device for detecting a head creature of the user An identification device and an eye biometric signal; and a processing device coupled to the user interface for performing the following steps: controlling the display device to display a page according to the head biometric signal; Identifying a signal, controlling the position of an indicator on the page; and when detecting an instruction of the user, controlling the electronic device to perform a specific function according to the position of the indicator on the page. The electronic device of claim 6, wherein the step of controlling the display device to display the page according to the biometric identification signal comprises: controlling, according to the biometric identification signal of the head, a screen displayed on the display device to a specific direction Scroll or move, or control the screen displayed on the display device to zoom in or out to display the page. The electronic device of claim 6, wherein the indicator comprises a cursor, a pattern, a frame line, or a specific color for indicating an item or region to be selected. The electronic device of claim 6, wherein the instruction comprises a static facial command or a dynamic combination of a plurality of facial actions. The electronic device of claim 6, wherein the form displays a form, the form includes a plurality of options, and when the face instruction is detected, controlling the electronic device according to the position of the indicator on the page The step of the device performing the specific function includes: when detecting the facial command, selecting one of the plurality of options indicated by the indicator according to the position of the indicator on the page. An interactive method for a user interface of an electronic device, the user interface comprising a display device, the interaction method comprising: detecting an eye biometric signal of a user, and identifying the eye according to the eye The signal controls an indication of a position on a page displayed by the display device; and detects a face command of the user, and when the face command is detected, according to the position of the indicator on the page Controlling the electronic device to perform a specific function; wherein the facial command is composed of a plurality of facial actions. The interaction method of claim 11, further comprising: detecting a biometric signal of the head of the user, and controlling the display device to display the page according to the biometric identification signal. The interactive method of claim 12, wherein the step of controlling the display device to display the page according to the head biometric signal comprises: controlling, according to the head biometric signal, a screen displayed on the display device to a specific direction Scroll or move, or control the screen displayed on the display device to zoom in or out to display the page. The interactive method of claim 11, wherein the indicator comprises a cursor, a pattern, a frame, or a specific color for indicating an item or region to be selected. The interactive method of claim 11, wherein the form displays a form, the form includes a plurality of options, and when the face instruction is detected, controlling the electronic device according to the position of the indicator on the page The step of the device performing the specific function includes: when detecting the facial command, selecting one of the plurality of options indicated by the indicator according to the position of the indicator on the page. An electronic device includes: a user interface for interacting with a user, the user bread comprises: a display device; and a sensing device for detecting an eye creature of the user An identification signal and a facial command, wherein the facial command is a combination of a plurality of facial motions; and a processing device coupled to the user interface for performing the following steps: a signal that controls a position of an indicator on a page displayed by the display device; and when the face command is detected, controls the electronic device to perform a specific function according to the position of the indicator on the page.