TW202121118A - Electronic device and method for identifying press on virtual keyboard - Google Patents

Abstract

An electronic device and a method for identifying press on a virtual keyboard are disclosed. A camera of the electronic device captures a plurality of consecutive images. A display of the electronic device sequentially displays the plurality of consecutive images, and displays the virtual keyboard including a plurality of virtual keys during displaying the plurality of consecutive images. A processor of the electronic device generates a key average differential map for each virtual key according to the plurality of consecutive images. Then, the processor analyzes all or part of the key average differential maps through an image classification model to determine which one of the virtual keys was pressed.

Description

Electronic device and method for recognizing pressing on virtual keyboard

The embodiment of the present invention relates to an image recognition method and electronic device. More specifically, the embodiments of the present invention relate to an electronic device and method for recognizing presses on a virtual keyboard.

A virtual keyboard is a non-physical keyboard created by a computer program, which is like a real space. Generally speaking, in order to identify the content input by the user, the traditional method will first define the coordinates of each virtual key on the virtual keyboard in the real space, and then according to the absolute value of the user’s fingertips in the real space. Position, to determine which virtual button the user's input is generated by. In order for the fingertip of the finger to touch the absolute position of any virtual key in the real space, the user must use a push method to move his finger. However, the pushing action (finger forward) is not coordinated with the general pressing action (finger down) used to operate the physical keyboard of the computer, which is not conducive to the user's gesture operation, thereby reducing the efficiency of using the virtual keyboard to input data. In addition, in order to avoid misjudgment, only when the user’s fingertip touches the absolute position of any virtual key in the real space, will it be judged that the virtual key is triggered, which makes the user’s operating range comparable Restriction, which in turn reduces the efficiency of using the virtual keyboard to input data. In view of this, how to improve the efficiency of using the virtual keyboard to input data will be relevant in the technical field. When it matters.

In order to solve at least the above-mentioned problems, embodiments of the present invention provide an electronic device for recognizing presses on a virtual keyboard, which may include a memory, a camera, a display, and a processor that are electrically connected to each other. The storage can be used to store an image classification model. The camera can be used to capture multiple consecutive images. The display can be used to sequentially display the multiple continuous images, and during the display of the multiple continuous images, display the virtual keyboard including multiple virtual keys. The processor can be used to: generate a button average difference map for each of the plurality of virtual keys according to the plurality of continuous images; and analyze all or part of the plurality of button average difference maps through the image classification model to determine Which one of the multiple virtual keys is pressed.

In order to solve at least the above-mentioned problems, embodiments of the present invention also provide a method for recognizing presses on a virtual keyboard. The method may include the following steps: capturing multiple continuous images from a camera; Sequentially display the multiple continuous images, and during the display of the multiple continuous images, display the virtual keyboard including multiple virtual keys; a processor, according to the multiple continuous images, for each of the multiple The virtual key generates a key average difference map; and the processor analyzes all or a part of the plurality of key average difference maps through an image classification model to determine which of the plurality of virtual keys is pressed.

In the embodiment of the present invention, because the average difference map of each button can reflect the changes of the corresponding virtual button in multiple consecutive images, the image classification model can identify the average difference map of each virtual button according to the button average difference map. Find out which virtual button was pressed by the finger. In other words, in the embodiment of the present invention, the actual pressing action of the finger can be recognized to determine which virtual key is pressed. Bottom) is consistent, so it is beneficial to the user's gesture operation. In addition, unlike the traditional method, in the embodiment of the present invention, the absolute position of the fingertip touching any virtual key in the real space is not required to identify the user's input, so it can be expanded to use Operator’s operating range. Accordingly, in the embodiment of the present invention, the efficiency of using the virtual keyboard to input data is significantly improved.

The above content is not intended to limit the present invention, but only briefly describes the technical problems that can be solved by the present invention, the technical means that can be adopted, and the technical effects that can be achieved, so that those with ordinary knowledge in the technical field to which the present invention belongs can have a preliminary understanding of the present invention. invention. According to the attached drawings and the content described in the following embodiments, those with ordinary knowledge in the technical field to which the present invention belongs can further understand the details of the various embodiments of the present invention.

As follows:

1‧‧‧Electronic device

11‧‧‧Processor

13‧‧‧Storage

15‧‧‧Camera

17‧‧‧Display

MD‧‧‧Image Classification Model

IM‧‧‧Image

2‧‧‧The process of identifying the pressing on the virtual keyboard by the electronic device

201, 203, 205, 207, 209, 211, 213‧‧‧Operation

USR‧‧‧User

X, Y, Z‧‧‧axis

KBD‧‧‧Virtual keyboard

VK, VK1, VK2‧‧‧Virtual buttons

SZ1, SZ2‧‧‧Image size

3‧‧‧The method of recognizing the press on the virtual keyboard

301, 303, 305, 307‧‧‧ steps

Figure 1 illustrates an electronic device that recognizes a press on a virtual keyboard according to some embodiments.

Fig. 2A illustrates how the electronic device shown in Fig. 1 recognizes a press on the virtual keyboard.

Figure 2B illustrates the environment in which the electronic device shown in Figure 1 captures images.

Figure 2C illustrates an image displayed on the display shown in Figure 1.

Figure 3 illustrates a method of recognizing presses on a virtual keyboard according to some embodiments.

Hereinafter, the present invention will be described through a number of embodiments, but these embodiments are not intended to limit the present invention to only be implemented according to the operation, environment, application, structure, process or steps. Elements that are not directly related to the present invention are not shown in the drawings, but may be implicit in the drawings. Yu Schema Here, the size of each element and the ratio between each element are only examples, and are not intended to limit the present invention. Except for special instructions, in the following content, the same (or similar) component symbols may correspond to the same (or similar) components. In the case of being realized, the number of each element described below may be one or more unless otherwise specified.

The terms used in this disclosure are only used to describe the embodiments and are not intended to limit the present invention. Unless the context clearly dictates otherwise, the singular form "one" is also intended to include the plural form. Terms such as "including" and "including" indicate the existence of the features, integers, steps, operations, elements, and/or elements, but do not exclude one or more other features, integers, steps, operations, elements, elements, and/or Or the existence of the aforementioned combination. The term "and/or" includes any and all combinations of one or more related listed items.

Figure 1 illustrates a schematic diagram of an electronic device that recognizes a press on a virtual keyboard according to some embodiments. The content shown in Figure 1 is only for illustrating the embodiments of the present invention, not for limiting the present invention. The electronic device 1 shown in FIG. 1 may be various electronic devices with computer functions, such as but not limited to: a server, a notebook computer, a tablet computer, a desktop computer, a mobile device, and so on. The electronic device 1 basically includes a processor 11, a storage 13, a camera 15, and a display 17 electrically connected to each other (directly or indirectly).

The processor 11 may include one or more microprocessors or microcontrollers with signal processing functions. A microprocessor or microcontroller is a special programmable integrated circuit, which has the capabilities of calculation, storage, output/input, etc., and can accept and process various coding instructions, so as to perform various logic operations and arithmetic operations, and output The corresponding calculation result. The processor 11 can store the complex images IM captured by the camera 15 in the memory 13 and perform various calculations. The processor 11 can also identify a virtual image according to various calculation results of the image IM. Press on the keyboard (detailed later).

The storage 13 may include various storage units. For example, the storage 13 may include a first-level memory (also called main memory or internal memory), which can be directly accessed by the processor 11. In addition to the first-level memory, in some embodiments, the storage 13 may also include a second-level memory (also called external memory or auxiliary memory), which communicates with the processor through the I/O channel of the memory. 11 link. The secondary memory may be a fixed storage entity such as a hard disk, an optical disk, etc. In addition to the first-level memory and the second-level memory, in some embodiments, the storage 13 may also include a portable storage device, such as a flash drive. In some embodiments, the storage 13 may also include a cloud storage unit. The storage 13 can store the data generated by the electronic device 1 itself and various data input to the electronic device 1 from the outside, such as the complex image IM captured by the camera 15, the image classification model MD, and the data required for the establishment of the image classification model MD Prototype, system, code or information, etc.

The camera 15 can be various devices with functions of dynamically capturing images and/or static capturing of images, such as but not limited to: digital cameras, video recorders, or various mobile devices with photography functions. In addition, the camera 15 can be connected to the processor 11 through a wired interface and/or a wireless interface. The camera 15 can be used to capture a plurality of continuous images IM related to the user's operation of the virtual keyboard for the processor 11 to analyze and calculate, and to recognize the pressing of the virtual keyboard according to the result of the analysis and calculation.

The display 17 may be various output devices, screens or panels with functions of displaying images and colors, such as but not limited to: liquid crystal display (LCD), light-emitting diode display (LED display) ), Organic LED display (OLED display), Micro LED display, Plasma Display Panel, Projection type Display...etc. The display 17 can be used to sequentially display a plurality of images IM captured by the camera 15 and simultaneously display a virtual keyboard including a plurality of virtual keys during the display of the images IM.

The following will take Figures 2A-2C as an example to illustrate the operation details of the electronic device 1 shown in Figure 1. Figure 2A illustrates how the electronic device 1 recognizes a press on a virtual keyboard KBD, and Figure 2B illustrates the electronic device 1 The device 1 captures the environment of multiple continuous images IM, and FIG. 2C illustrates the image displayed on the display 17. The content shown in Figures 2A-2C is only for exemplifying the embodiments of the present invention, not for limiting the present invention.

Referring to Figure 2A, in the process 2 of the electronic device 1 recognizing the press on the virtual keyboard KBD, first, the camera 15 of the electronic device 1 captures a plurality of continuous images IM (labeled as operation 201), and the images IM can be It is stored in the storage 13. Referring to FIG. 2B, the camera 15 may be set on the electronic device 1 with its lens facing the user's USR hand to capture multiple continuous images IM for the user's USR hand posture in a time interval. . In some embodiments, when the lens of the camera 15 is still facing the user's USR hand, the camera 15 may not be installed on the electronic device 1, but connected to the electronic device 1 through a wired or wireless method. .

As shown in FIG. 2C, whenever the camera 15 captures an image IM, the processor 11 can display the image IM on the display 17. In addition, while the display 17 sequentially displays a plurality of continuous images IM, a virtual keyboard KBD including a plurality of virtual keys VK can be displayed at the same time. According to different needs, the virtual keyboard KBD can have different forms and different numbers of virtual keys VK, and it is not limited to the content shown in Figure 2C. Taking Fig. 2B as an example, the display 17 may be arranged on the electronic device 1 for the user USR to view. However, in some embodiments, when the user USR can still view the display content of the display 17, the display 17 may not be set on the electronic device 1, but can communicate with the electronic device through a wired or wireless method. 1 link.

The user USR can confirm the relative position of his hand and the virtual keyboard KBD according to the display screen of the display 17, and then move the position of the hand or the position of the finger to press a certain virtual key VK. Therefore, through the camera 15 and the display 17, the user USR can operate the virtual keyboard KBD without touching any physical equipment or components.

Continuing to refer to FIG. 2A, after completing operation 201, the processor 11 may generate a key average difference map corresponding to the k-th virtual key VK based on the position of the k-th virtual key VK according to the multiple continuous images IM (Labeled as operation 203), where k is a positive integer between 1 and j, and j is the total number of virtual keys VK included in the virtual keyboard KBD.

In operation 203, the processor 11 may first determine whether the multiple images IM are grayscale images, and if not, the processor 11 converts the multiple images IM into grayscale images. Then, in some embodiments, the processor 11 may crop the multiple continuous gray-scale images based on the position of the k-th virtual key VK to obtain a group of continuous gray-scale images corresponding to the k-th virtual key VK. Button images, where the group of button images have the same image size. Taking Figure 3C as an example, the group of key images corresponding to the virtual key VK1 have the same size SZ1, and the size SZ1 is larger than the image size of the virtual key VK1, and the group of key images corresponding to the virtual key VK2 have the same size The size SZ2, and the size SZ2 is larger than the image size of the virtual button VK2.

Then, the processor 11 may generate a key average difference map corresponding to the k-th virtual key VK according to the group of continuous key images, wherein the image size of the key average difference map is the same as the image size of the group of continuous key images , And greater than or equal to the image size of the k-th virtual key VK. For example, the processor 11 may calculate an average difference value (standard deviation) for each group of pixel values corresponding to the same position in the group of consecutive button images, and combine these average difference values into the kth according to the corresponding pixel positions. The average difference map of the virtual keys VK.

In addition, in some other embodiments, the processor 11 may first calculate an image average difference map based on the multiple continuous gray-scale images, that is, the processor 11 first calculates an image average difference map based on the multiple continuous gray-scale images. An average difference value is calculated for each group of pixel values corresponding to the same position in the corresponding pixel position, and these average difference values are combined into the image average difference map according to the corresponding pixel positions, and then the image average difference map is stored in the storage 13. Then, the processor 11 cuts the image average difference map based on the position of the k-th virtual button VK to obtain a button average difference map corresponding to the k-th virtual button VK, wherein the image size of the button average difference map is It can be greater than or equal to the image size of the k-th virtual key VK. In these embodiments, since the processor 11 has already stored the image average difference map in the storage 13 when the operation 203 is executed for the first time, when operation 203 needs to be executed again, the image average difference map does not need to be repeatedly calculated. Only by cutting the image average difference map based on the positions of other virtual buttons VK, the corresponding button average difference map can be generated.

In some embodiments, the length and width of each key average difference map may be three times the length and width of the corresponding virtual key VK1. In the case that the image size of a key average difference map exceeds the boundary of the image IM, the excess pixel values can be set to a preset value (for example, "0").

Continuing to refer to FIG. 2A, in some embodiments, after completing operation 203, the processor 11 may perform operations 205 and 207 to determine whether to analyze the key corresponding to the k-th virtual key through the image classification model MD Average difference graph. In detail, in operation 205, the processor 11 may binarize the button average difference map to generate a binarized button average difference map. Taking a grayscale image as an example, the processor 11 may adjust the pixel values in the button average difference map that are greater than a binarization threshold (for example, "127") to the upper extreme value "1", and the average difference of the button In the figure, those pixel values that are not greater than the binarization threshold value are adjusted to the lower extreme value "0", thereby producing Generate the average difference map of the binarized button.

Then, in operation 207, the processor 11 may determine the average value of the pixel values of the binarized button average difference map (that is, the value generated by dividing the sum of the pixel values of the binarized button average difference map by the number of pixels) Is it greater than a momentum threshold (for example, "0.5"). If yes, it means that the corresponding virtual key VK may have just been pressed by the user USR, so operation 209 is continued; if not, it means that the corresponding virtual key VK may not have been pressed by the user USR, so the current k value is increased One, and then perform operation 203 again (that is, generate the key average difference map of the next virtual key). Through operations 205 and 207, the virtual keys that may not be pressed can be roughly eliminated first, so as to save the number of times of performing operation 209, thereby reducing the amount of calculation and improving the calculation efficiency. In other words, by judging whether the average pixel value of the binarized button average difference map of each virtual button VK is greater than a momentum threshold, it can be determined whether it is necessary to analyze the average button difference of the virtual button VK through the image classification model MD. Figure.

In some other embodiments, after the operation 203 is completed, the processor 11 may not perform the operation 205, and the operation 207 may be replaced with: the processor 11 determines whether the key average difference map corresponding to the k-th virtual key VK is Whether the sum or average of the pixels is greater than a momentum threshold, and use this to determine whether to analyze the button average difference map.

In some embodiments, after completing operation 203, the processor 11 may not perform operations 205 and 207, but directly perform operation 209.

In operation 209, the processor 11 can analyze the key average difference map corresponding to the k-th virtual key through the image classification model MD stored in the storage 13. In detail, the processor 11 may input the key average difference map corresponding to the k-th virtual key into the image classification model MD generated by machine training in advance, and determine the k-th virtual key according to the classification result of the image classification model MD. Whether each virtual key is pressed.

In some embodiments, the image classification model MD may be established by using multiple non-pressed button average difference maps and multiple pressed button average difference maps as a training data set for machine learning. The machine learning can be implemented by means such as but not limited to Convolutional Neural Network (CNN), Support Vector Machine Model (Support Vector Machine Model), Recurrent Neural Network Model (Recurrent Neural Network Model), etc. For example, a lightweight six-layer convolutional layer combined with a fully connected layer and a SoftMax classification layer can be used in a convolutional neural network to build an image classification model MD.

In some embodiments, the image classification model MD can be established by the electronic device 1 itself. For example, a virtual key VK on the virtual keyboard KBD may be displayed repeatedly on the display 17 randomly or according to a preset rule, and a value between 1 and 10 may be displayed randomly or according to a preset rule. The number that represents the user's USR finger (for example, "1" represents the pinky of the left hand, "2" represents the ring finger of the left hand, ..., "10" represents the pinky of the right hand, etc.). Whenever the display 17 displays a virtual key VK and a value between 1 and 10, the user USR can move the finger designated by the value to the position of the virtual key VK, and then press it for the camera 15 to capture A group of pressed button images corresponding to the virtual button VK. Whenever the camera 15 captures a group of pressed button images, the processor 11 can calculate the average difference map of the pressed buttons corresponding to a virtual key VK according to the group of pressed button images. Similarly, whenever the display 17 displays a virtual key VK and a value between 1 and 10, the user USR can move the finger designated by the value to the position of the virtual key VK, but does not press it, so that The camera 15 captures a set of non-press button images corresponding to the virtual button VK. Whenever the camera 15 captures a set of non-press button images, the processor 11 can calculate a virtual button based on the set of non-press button images VK corresponding non-press button average difference graph. After generating enough pressed button average difference maps and non-pressed button average difference maps, the processor 11 can use these button average difference maps as a training data set for machine learning to build an image classification model MD. According to different requirements, the number of non-press button average difference graphs and the number of pressed button average difference graphs are variable. For example, in some embodiments, the number of non-pressed button average difference maps may be at least three times the number of pressed button average difference maps.

In some embodiments, the image classification model MD can also be pre-built by an external device and stored in the storage 13 of the electronic device 1 in advance.

Continuing to refer to FIG. 2A, after the processor 11 completes operation 209, it can then determine whether the classification result of the image classification model MD is "0" (labeled as operation 211). If the classification result is "0", it means that the k-th virtual key VK has not been pressed, so the current k value is increased by one, and operation 203 is performed again (that is, the average key difference map of the next virtual key VK is generated) . If the classification result is not "0", it means that the k-th virtual button VK was pressed by a finger of the user USR, so the processor 11 can generate the corresponding content of the k-th virtual button VK (marked as operation 213) .

Figure 3 illustrates a schematic diagram of a method for recognizing a press on a virtual keyboard according to some embodiments. The content shown in Figure 3 is only for illustrating the embodiments of the present invention, not for limiting the present invention.

Referring to Figure 3, a method 3 for recognizing presses on a virtual keyboard may include the following steps: a camera captures multiple continuous images (labeled as step 301); a display sequentially displays the multiple continuous images During the display of the multiple continuous images, the virtual keyboard (labeled as step 303) containing multiple virtual keys is displayed; For successive images, a button average difference map (labeled as step 305) is generated for each of the plurality of virtual buttons; and the processor analyzes all or part of the plurality of button average difference maps through an image classification model to It is determined which one of the plurality of virtual keys is pressed (marked as step 307).

The order of the steps shown in Figure 3 is not a limitation, and the order of the steps shown in Figure 3 can be arbitrarily adjusted if it can still be implemented.

In some embodiments, in step 307, the image classification model is one of a convolutional neural network model, a support vector machine model, and a recurrent neural network model.

In some embodiments, in addition to steps 301 to 307, the method 3 for recognizing presses on a virtual keyboard may also include the following steps: whenever one of the plurality of key average difference maps is generated, binarize the key average Difference map to generate a binarized button average difference map; and determine whether the average pixel value of the binarized button average difference map is greater than a preset threshold, so as to determine whether to analyze the corresponding difference through the image classification model Corresponding button average difference graph.

In some embodiments, the image classification model is established by using multiple non-pressed button average difference maps and multiple pressed button average difference maps as a training data set for machine learning.

In some embodiments, the size of the average difference map of each of the plurality of buttons is larger than the image size of the corresponding virtual button.

The method 3 of recognizing a press on a virtual keyboard basically includes embodiments corresponding to all the above-mentioned embodiments of the electronic device 1. Therefore, in addition to the above-mentioned embodiment of the method 3 for recognizing a press on a virtual keyboard, the method 3 for recognizing a press on a virtual keyboard may also include other embodiments, and those with ordinary knowledge in the technical field of the present invention can follow the above Text for electronics The description of the device 1 directly understands these other embodiments, so it is not repeated here.

The above-mentioned embodiments are only examples to illustrate the present invention, but not to limit the present invention. Any other embodiments resulting from modification, change, adjustment, and integration of the above-mentioned embodiments, as long as those with ordinary knowledge in the technical field of the present invention are not difficult to think of, are covered by the protection scope of the present invention. The scope of protection of the present invention is subject to the scope of the patent application.

3‧‧‧The method of recognizing the press on the virtual keyboard

301, 303, 305, 307‧‧‧ steps

Claims (10)

An electronic device for recognizing presses on a virtual keyboard, including: A storage for storing an image classification model; A camera to capture multiple consecutive images; A display for displaying the multiple continuous images in sequence, and displaying the virtual keyboard including multiple virtual keys during the display of the multiple continuous images; and A processor is electrically connected to the storage, the camera and the display, and is used to: According to the multiple continuous images, a key average difference map is generated for each of the multiple virtual keys; and The image classification model is used to analyze all or part of the average difference map of the plurality of buttons to determine which one of the plurality of virtual buttons is pressed. The electronic device according to claim 1, wherein the image classification model is one of a convolutional neural network model, a support vector machine model, and a recurrent neural network model. The electronic device according to claim 1, wherein the processor is further used for: Whenever one of the plurality of button average difference maps is generated, binarize the button average difference map to generate a binarized button average difference map; and It is determined whether the average value of the pixel values of the binarized button average difference map is greater than a preset threshold value, so as to determine whether to analyze the corresponding button average difference map through the image classification model. The electronic device according to claim 1, wherein the image classification model is established by using multiple non-press button average difference maps and multiple pressed button average difference maps as a training data set for machine learning. The electronic device according to claim 1, wherein the size of each of the plurality of button average difference graphs is larger than and The image size of the corresponding virtual key. A method of recognizing presses on a virtual keyboard includes: Capture multiple continuous images from one camera; A display sequentially displays the multiple continuous images, and displays the virtual keyboard including multiple virtual keys during the display of the multiple continuous images; A processor generates a key average difference map for each of the plurality of virtual keys according to the plurality of continuous images; and The processor analyzes all or a part of the average difference map of the plurality of keys through an image classification model to determine which one of the plurality of virtual keys is pressed. The method according to claim 6, wherein the image classification model is one of a convolutional neural network model, a support vector machine model, and a recurrent neural network model. The method according to claim 6, further comprising: Whenever one of the plurality of button average difference maps is generated, binarize the button average difference map to generate a binarized button average difference map; and It is determined whether the average value of the pixel values of the binarized button average difference map is greater than a preset threshold value, so as to determine whether to analyze the corresponding button average difference map through the image classification model. The method according to claim 6, wherein the image classification model is established by using multiple non-pressed button average difference maps and multiple pressed button average difference maps as a training data set for machine learning. The method according to claim 6, wherein the size of the average difference map of each of the plurality of buttons is larger than the image size of the corresponding virtual button.

Images