JP2020177363A - Estimation system and estimation device - Google Patents

Abstract

To provide an estimation system and an estimation device capable of improving the accuracy of estimating a key input.SOLUTION: An estimation system for estimating a character string inputted by using a virtual keyboard includes: acquisition means for acquiring object information including angle information on the angle of a user's finger on the basis of photographic information obtained by photographing movement of depressing the virtual keyboard by the user; a reference database storing the degree of relation of three stages or more between previously acquired past object information and reference information including character strings inputted by the past object information; evaluation means for acquiring evaluation information including a first degree of relation of three stages or more between the object information and the reference information with reference to the reference database; and generation means for generating an estimation result on the basis of the evaluation information.SELECTED DRAWING: Figure 1

Description

The present invention relates to an estimation system and an estimation device that estimate a character string input using a virtual keyboard.

Conventionally, as a technique for controlling key input without using a physical keyboard, for example, an information processing device of Patent Document 1 has been proposed.

In Patent Document 1, a hand region is extracted from an image input by a photographing unit that captures a hand, a reference line determining unit that determines a predetermined line in the image as a pressing reference line based on the hand region, and a hand. A determination unit that determines that the hand is pressed when the bottom of the area exceeds the pressing reference line, and a first position determination unit that determines the position in the depth direction based on the aspect ratio of the hand area when the determination unit determines that the hand is pressed. Based on the position of the lowermost part of the hand area when it is determined by the determination unit to be pressed, the second position determination unit that determines the lateral position, and the determined depth and lateral positions are used for input. An information processing apparatus including an input key determination unit for determining a key is disclosed.

International Publication No. 2012/176315

Here, for example, in an information processing device such as Patent Document 1, an input command is estimated by a threshold value based on an image taken from one direction by one camera. Therefore, it is necessary to perform key input according to the threshold value, and there is a concern that variation in pressing depth cannot be taken into consideration, and it is difficult to improve the estimation accuracy of key input. Therefore, improving the accuracy of estimating key input is an issue.

Therefore, the present invention has been devised in view of the above-mentioned problems, and an object of the present invention is to provide an estimation system and an estimation device capable of improving the accuracy of estimating key input. It is in.

It is an estimation system that estimates a character string input using a virtual keyboard, and based on shooting information obtained by photographing the action of a user pressing the virtual keyboard, target information including angle information regarding the angle of the user's finger is obtained. A reference database in which three or more levels of association between the acquisition means to be acquired, the past target information acquired in advance, and the reference information including the character string input by the past target information is stored, and the above. An evaluation means that refers to a reference database and acquires evaluation information including a first degree of association between the target information and the reference information in three or more stages, and a generation means that generates an estimation result based on the evaluation information. It is characterized by having.

In the estimation system according to the second invention, in the first invention, the acquisition means includes a shooting information acquisition means for acquiring the shooting information and a coordinate data acquisition means for acquiring the coordinate data of the user's finger from the shooting information. It is characterized by having an angle information acquisition means for acquiring the angle information from the coordinate data.

In the second invention, the estimation system according to the third invention includes the three-dimensional image data generated by using the three-dimensional camera, and the coordinate data acquisition means obtains the coordinates from the three-dimensional image data. It is characterized by acquiring data.

The estimation system according to the fourth invention is characterized in that, in any one of the first to third inventions, the target information and the past target information include optical flow information indicating the movement of the user's finger. ..

In the fourth invention, the estimation system according to the fifth invention includes the RGB image generated by using the RGB camera, and the acquisition means acquires the optical flow information from the plurality of the RGB images. It is characterized by having an optical flow information acquisition means.

The estimation system according to the sixth invention is characterized in that, in any one of the first to fifth inventions, the target information and the past target information include keyboard information indicating the type of the virtual keyboard.

The estimation system according to the seventh invention is characterized in that, in any one of the first to sixth inventions, the reference database is constructed by machine learning.

The estimation device according to the eighth invention is an estimation device that estimates a character string input by using a virtual keyboard, and is based on shooting information obtained by photographing an action of the user pressing the virtual keyboard, and is a finger of the user. Three or more levels of association between the acquisition unit that acquires the target information including the angle information regarding the angle, the past target information acquired in advance, and the reference information including the character string input by the past target information. In the evaluation unit that refers to the reference database in which is stored and the reference database, and acquires evaluation information including the first degree of association between the target information and the reference information in three or more stages, and the evaluation information. It is characterized by including a generation unit that generates an estimation result based on the above.

According to the first to seventh inventions, the evaluation means refers to the reference database and acquires the evaluation information including the first degree of association between the target information and the reference information in three or more stages. Therefore, it is possible to generate an estimation result based on the past results, and it is possible to estimate the key input in consideration of the variation in the pressing depth. This makes it possible to improve the accuracy of estimating the key input.

Further, according to the first to seventh inventions, the target information includes angle information. Therefore, as compared with the case of calculating the position of the hand from one image, the positional relationship of the hand with respect to the virtual keyboard can be easily acquired regardless of the characteristics of the hand that are different for each user. This makes it possible to acquire the information necessary for estimating the key input with high accuracy.

Further, according to the second invention, the coordinate data acquisition means acquires the coordinate data from the shooting information. Further, the angle information acquisition means acquires angle information from the coordinate data. Therefore, the amount of data to be evaluated can be reduced. As a result, when the target information is stored, it is possible to suppress an increase in the storage capacity. In addition, the parameters required for evaluation can be suppressed, and the estimation accuracy can be further improved.

Further, according to the third invention, the shooting information includes three-dimensional image data generated by using a three-dimensional camera. Therefore, when acquiring the angle information, a conventional three-dimensional camera can be used without providing a new camera or the like. This makes it possible to easily prepare an environment for estimating key input.

Further, according to the fourth invention, the target information and the past target information include optical flow information. Therefore, by combining the optical flow information and the angle information, it is possible to consider the movement at the time of key input for each user. This makes it possible to further improve the accuracy of estimating the key input.

Further, according to the fifth invention, the optical flow information acquisition means acquires optical flow information from a plurality of RGB images. Therefore, when acquiring optical flow information, a conventional RGB camera can be used without providing a new camera or the like. This makes it possible to easily prepare an environment for improving the estimation accuracy of key input.

Further, according to the sixth invention, the target information and the past target information include keyboard information. Therefore, even when a virtual keyboard having a different key layout or the like is used, it is possible to estimate the key input with high accuracy.

Further, according to the seventh invention, the reference database is constructed by machine learning. Therefore, it is possible to estimate the key input excluding the subjectivity as compared with the case of setting the threshold value for the pressing depth calculated from the image.

According to the eighth invention, the evaluation unit refers to the reference database and acquires the evaluation information including the first degree of association between the target information and the reference information in three or more stages. Therefore, it is possible to generate an estimation result based on the past results, and it is possible to estimate the key input in consideration of the variation in the pressing depth. This makes it possible to improve the accuracy of estimating the key input.

Further, according to the eighth invention, the target information includes angle information. Therefore, as compared with the case of calculating the position of the hand from one image, the positional relationship of the hand with respect to the virtual keyboard can be easily acquired regardless of the characteristics of the hand that are different for each user. This makes it possible to acquire the information necessary for estimating the key input with high accuracy.

FIG. 1 (a) is a schematic diagram showing an example of an estimation system in the present embodiment, and FIG. 1 (b) is a schematic diagram showing another example of the estimation system in the present embodiment. FIG. 2 is a schematic diagram showing an example of the operation of the estimation system. FIG. 3 is a schematic diagram showing an example of an acquisition means for acquiring angle information and optical flow information. FIG. 4A is a schematic diagram showing an example of the configuration of the estimation device according to the present embodiment, and FIG. 4B is a schematic diagram showing an example of the function of the estimation device according to the present embodiment. FIG. 5 is a schematic diagram showing an example of a reference database according to the present embodiment. FIG. 6 is a schematic view showing a first modification of the reference database in the present embodiment. FIG. 7 is a schematic view showing a second modification of the reference database in the present embodiment. FIG. 8 is a flowchart showing an example of the operation of the estimation system according to the present embodiment.

Hereinafter, an example of the estimation system and the estimation device according to the embodiment to which the present invention is applied will be described with reference to the drawings.

An example of the estimation system 100 and the estimation device 1 in the present embodiment will be described with reference to FIG.

The estimation system 100 in this embodiment has an estimation device 1 as shown in FIG. 1A, for example. The estimation device 1 is connected to the user terminal 3 via, for example, the communication network 4. The estimation system 100 estimates a character string input using the virtual keyboard 8. In the estimation system 100, the virtual keyboard 8 is projected by the user terminal 3, and the virtual keyboard 8 may be projected by the estimation device 1 as shown in FIG. 1B, for example, and the virtual keyboard may be projected by another terminal or the like. 8 may be projected.

In the estimation system 100, for example, as shown in FIG. 2, the estimation device 1 acquires the target information and generates an estimation result for the target information. The target information includes angle information regarding the angle of the user's finger, and may include, for example, shooting information. The shooting information includes image data obtained by shooting an operation of the user pressing the virtual keyboard 8 and position information of the user's hand in the image data. The target information may include, for example, optical flow information indicating the movement of the user's finger. The target information may be generated by the user terminal 3 or, for example, the estimation device 1.

The image data may be generated by using a known camera provided in the user terminal 3 or the estimation device 1, or may be generated by using a known camera independent of the user terminal 3 or the estimation device 1. The image data includes an image of the user's hand pressing one key, and may include, for example, a plurality of images (for example, a moving image) of the movement of the user's hand before and after pressing at least one key. The image data may be, for example, a moving image of the movement of the user's finger, in addition to the image of the user's finger. The image data includes information about the keyboard such as the shape, projection size, key height, and key layout of the virtual keyboard 8 (equivalent to keyboard information described later).

The image data includes three-dimensional image data, for example, RGB image data and infrared image data. The three-dimensional image data is generated using a known three-dimensional camera (3D camera). As the three-dimensional camera, for example, a plurality of RGB cameras may be used instead. The RGB image data is generated by using an RGB camera corresponding to a visible light region (for example, 400 nm or more and 800 nm or less). Infrared image data is generated using an infrared camera corresponding to an infrared light region (for example, 800 nm or more and 1,000 μm or less). The RGB image data and the infrared image data may be generated using, for example, one multispectral camera corresponding to the visible light region to the infrared light region.

The position information is generated using, for example, a known distance measuring camera that measures the distance from the object. The position information may be generated by using a distance measuring camera provided in the user terminal 3 or the estimation device 1, or may be generated by using a distance measuring camera independent of the user terminal 3 or the estimation device 1.

The position information has distance data indicating the distance between the distance measuring camera and the user's hand (object) captured by the image data. The distance data indicates the distance between the distance measuring camera and at least one of the preset user's hands. The distance data may indicate, for example, the distance between the distance measuring camera and each fingertip of the user. The fingertips of the user indicate parts that are expected to come into contact with the virtual keyboard 8. For example, a total of 10 locations corresponding to the fingertips of the right hand and the fingertips of the left hand are the targets of the distance data.

The distance data is generated in the same period as when the image data is taken, for example. The distance data may be generated only during the period before and after the user presses the virtual keyboard 8, for example. The distance data is generated as numerical data and may be represented by a vector or a matrix.

The position information may include, for example, coordinate data. The coordinate data is shown (generated) by, for example, a coordinate space using the x-axis, y-axis, and z-axis with respect to the position of the distance measuring camera, and may be acquired from, for example, three-dimensional image data. Like the distance data, the coordinate data may indicate the coordinates of at least one place of the user's hand, for example, the coordinates of each fingertip of the user. The coordinate data is generated in the same period as when the image data is taken, for example. The coordinate data may be generated only during the period before and after the user presses the virtual keyboard 8, for example. The coordinate data is generated as numerical data and may be represented by a vector or a matrix.

The angle information is acquired based on the three-dimensional image data, for example, as shown in FIG. The angle information may be acquired from the coordinate data acquired from the three-dimensional image data, or may be acquired from the coordinate data generated by, for example, a distance measuring camera. The angle information may be obtained, for example, by calculating the angle from the coordinates of three or more points centered on the coordinates at the joint portion of the coordinates of each finger. A known technique can be used as a method for acquiring angle information.

The optical flow information is acquired based on RGB image data, for example, as shown in FIG. The optical flow information may be acquired from a plurality of RGB images or moving images, or may be acquired from, for example, a plurality of coordinate data having different acquisition times. The optical flow information indicates, for example, the time course of the position at each fingertip. The optical flow information may be acquired in an image format or, for example, in a matrix (vector) format. A known technique can be used as a method for acquiring optical flow information.

The projection unit, the three-dimensional camera, the RGB camera, the infrared camera, and the distance measuring camera for projecting the virtual keyboard 8 described above may be provided in the same terminal (for example, the estimation device 1 or the user terminal 3). Further, the above-mentioned image data and position information may be generated by using a known camera such as Kinect (registered trademark) or a multispectral camera. In this case, for example, a camera having two or more eyes may be used.

The target information may include, for example, keyboard information. The keyboard information indicates the type of virtual keyboard 8 used by the user. The keyboard information includes, for example, an array of keys, a size of each key, a key height, and the like, in addition to the projection size of the virtual keyboard 8.

The target information may include, for example, linguistic information. The language information indicates the type of language that the user inputs using the virtual keyboard 8. In the estimation system 100, the input character string may be estimated based on the language information. In this case, the key input can be estimated with high accuracy without depending on the language used by the user. In addition, it is possible to generate an estimation result in consideration of typos and the like.

The estimation result shows the result of estimating the content corresponding to the input key when the user presses the virtual keyboard 8. As the estimation result, a character string is generated, and for example, a control command of the estimation device 1 or the user terminal 3 is generated. In the estimation system 100, the accuracy of the estimation result can be improved by increasing the types of information included in the target information. In particular, by combining the angle information and the optical flow information, it is possible to capture the movement of the user's finger with high accuracy.

(Estimator 1)
Next, an example of the estimation device 1 in the present embodiment will be described with reference to FIG. FIG. 4A is a schematic diagram showing an example of the configuration of the estimation device 1 in the present embodiment, and FIG. 4B is a schematic diagram showing an example of the function of the estimation device 1 in the present embodiment.

As the estimation device 1, for example, an electronic device such as a personal computer (PC) is used, and for example, an electronic device such as a smartphone, a tablet terminal, a wearable terminal, an IoT (Internet of Things) device, a Raspberry Pi (registered trademark), or the like. A single board computer may be used. As shown in FIG. 4A, for example, the estimation device 1 includes a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and a storage unit. It includes 104 and I / F 105-107. Each configuration 101-107 is connected by an internal bus 110.

The CPU 101 controls the entire estimation device 1. The ROM 102 stores the operation code of the CPU 101. The RAM 103 is a work area used during the operation of the CPU 101. The storage unit 104 stores various information such as shooting information and target information. As the storage unit 104, for example, in addition to an HDD (Hard Disk Drive), a data storage device such as an SSD (solid state drive) or a floppy disk is used. For example, the estimation device 1 may have a GPU (Graphics Processing Unit) (not shown). Having a GPU enables faster arithmetic processing than usual.

The I / F 105 is an interface for transmitting and receiving various information to and from the user terminal 3 and the server 2 via a communication network 4 such as the Internet. The I / F 106 is an interface for transmitting / receiving information to / from the input portion 108. For example, a keyboard is used as the input portion 108, and the administrator of the estimation device 1 inputs various information or control commands of the estimation device 1 via the input portion 108. The I / F 107 is an interface for transmitting and receiving various information to and from the output portion 109. The output unit 109 outputs various information stored in the storage unit 104, the processing status of the estimation device 1, and the like. A display is used as the output portion 109, and for example, a touch panel type may be used.

For example, when the output portion 109 has a projection unit for projecting the virtual keyboard 8, the input portion 108 has various cameras for generating target information (for example, the above-mentioned three-dimensional camera, RGB camera, infrared camera, and distance). You may have at least one of the measuring cameras). Further, when a terminal such as a smartphone or a tablet is used as the estimation device 1, a known projection device (projection unit) provided is used as a part of the output unit 109, and a known camera (imaging unit) provided is used. May be used as part of the input portion 108.

<Reference database>
In the storage unit 104, for example, a reference database in which past target information, reference information, and association degree acquired in advance are stored is stored. The past target information indicates the same type of information as the above-mentioned target information, and includes past angle information, for example, at least one of past optical flow information, past image data, past position information, and past keyboard information. May include. The reference information includes a character string input by the past target information, and includes, for example, a control command of the estimation device 1 or the user terminal 3. The degree of association indicates the degree of relationship between the past target information and the reference information.

As shown in FIG. 5, for example, the reference database stores three or more levels of association between the past target information and the reference information. The degree of association is indicated by three or more stages such as percentage, 10 stages, or 5 stages, and is indicated by, for example, line characteristics (for example, thickness). The reference database is formed by an algorithm that can calculate the degree of association using, for example, past target information and reference information. The past target information and the reference information have a plurality of data, and the relationship between each past target information and each reference information is linked by the degree of association.

For example, "information A" included in the past target information indicates a degree of association "80%" with "reference A" included in the reference information, and is between "reference B" included in the reference information. The degree of association is "15%". That is, the "degree of association" indicates the degree of connection between each data, and the higher the degree of association, the stronger the connection of each data.

The past target information is stored in the reference database in the form of an image (a plurality of images showing changes over time, a moving image may be used) or a character string, or in a format such as a numerical value, a matrix (vector), or a histogram. It may be memorized by. Further, the reference information is stored in the reference database in a format such as a character string, or may be stored in a format such as a numerical value, a matrix (vector), or a histogram. That is, the "information A", "information B", and "information C" shown in FIG. 5 are target information acquired in the past (for example, in addition to angle information, optical flow information, RGB image data, infrared image data, distance). Data, coordinate data, and at least one of keyboard information) indicates data stored in a format such as an image format or a matrix, and "reference A", "reference B", and "reference C" have been acquired in the past. The data stored in the format of a character string or a format corresponding to the estimation result is shown.

The reference database is constructed using, for example, machine learning. In this case, the degree of association is calculated using machine learning. For machine learning, for example, deep learning is used. The reference database is constructed, for example, in a neural network, in which case the degree of association may be indicated by hidden layers and weight variables. That is, in the estimation system 100, a reference database constructed by machine learning can be used, and in this case, the degree of association between the past target information having a plurality of data and the reference information is complicated. It can be shown with higher accuracy. This makes it possible to improve the accuracy of the estimation result.

As shown in FIG. 6, for example, the past target information may be stored in the reference database by dividing the past angle information and the past optical flow information. In this case, the degree of association is calculated based on the relationship between the combination of the past angle information and the past optical flow information and the reference information. In addition to the above, the past target information may be stored in the reference database by dividing at least one of the past image data and the past position information.

For example, the combination of "angle A" included in the past angle information and "flow A" included in the past optical flow information indicates a degree of association "90%" with "reference A", and "reference". The degree of association with "B" is "20%". In this case, the past angle information and the past optical flow information can be stored independently. Therefore, when generating the estimation result, it is possible to improve the accuracy and expand the range of options.

The past image data may be stored in the reference database by dividing the past RGB image data and the past infrared image data, for example. Further, the past position information may be stored in the reference database by dividing the past distance data and the past coordinate data, for example.

The past target information may include synthetic data and similarity, as shown in FIG. 7, for example. The synthetic data is represented by three or more levels of similarity with past angle information or past optical flow information. The composite data is stored in the reference database in the form of an image, a character string, or the like, or may be stored in a format such as a numerical value, a matrix, or a histogram.

FIG. 4B is a schematic diagram showing an example of the function of the estimation device 1. The estimation device 1 includes an acquisition unit 11, a storage unit 12, an evaluation unit 13, a generation unit 14, and an output unit 15, and may include, for example, an update unit 16. Each function shown in FIG. 4B is realized by the CPU 101 executing a program stored in the storage unit 104 or the like using the RAM 103 as a work area, and may be controlled by, for example, artificial intelligence.

<Acquisition unit 11>
The acquisition unit 11 acquires the target information. The acquisition unit 11 acquires target information from an external terminal such as a user terminal 3 via the communication network 4 and the I / F 105, and for example, an imaging unit (for example, the above-mentioned three-dimensional camera, RGB camera, etc.) included in the input unit 108 Target information is acquired from at least one of an infrared camera and a distance measurement camera). The acquisition unit 11 may acquire the shooting information and acquire the angle information based on the shooting information as the target information, or may collectively acquire, for example, the angle information or the like as the target information. The frequency and cycle for the acquisition unit 11 to acquire the target information are arbitrary.

The acquisition unit 11 acquires coordinate data from, for example, the acquired shooting information, and acquires angle information from the coordinate data. As a result, the acquisition unit 11 acquires the target information including the angle information. The acquisition unit 11 acquires optical flow information from, for example, a plurality of acquired RGB images. As a result, the acquisition unit 11 acquires the target information including the angle information and the optical flow information. The acquisition unit 11 can acquire the angle information and the optical flow information by using the above-mentioned known technique.

The acquisition unit 11 receives various information transmitted to the estimation device 1. The acquisition unit 11 receives various information such as target information transmitted from an external terminal such as the user terminal 3 via the communication network 4 and the I / F 105.

<Memory unit 12>
The storage unit 12 retrieves various information such as a reference database stored in the storage unit 104 as needed. The storage unit 12 stores various information acquired or generated by the configurations 11 and 13 to 16 in the storage unit 104.

<Evaluation unit 13>
The evaluation unit 13 refers to the reference database and acquires evaluation information including the first degree of association between the target information and the reference information. When the evaluation unit 13 refers to the reference database shown in FIG. 4, for example, the evaluation unit 13 selects past target information (for example, “information A”: past first target information) that is the same as or similar to the information contained in the target information. select. As the past target information, information that partially matches or completely matches the target information is selected, and for example, information that is similar (including the same concept or the like) is selected. When the target information is represented by a numerical value such as a matrix, the numerical range included in the selected past target information may be set in advance.

The evaluation unit 13 selects the reference information associated with the selected past first target information and the degree of association (first degree of association) between the selected past target information and the reference information, and acquires it as evaluation information. .. A part of the first degree of association is selected from the degree of association, and may be calculated by the evaluation unit 13.

For example, the evaluation unit 13 selects the reference information "reference A" associated with the first target information "information A" and the first degree of association "80%" between "information A" and "reference A". Obtained as evaluation information. The reference information and the first degree of association may include a plurality of data. In this case, in addition to the above-mentioned "reference A" and "80%", the reference information "reference B" associated with the first target information "information A" and between "information A" and "reference B" The first degree of association "15%" may be selected, and "reference A" and "80%", and "reference B" and "15%" may be acquired as evaluation information.

The evaluation information may include target information. Further, the first degree of association is indicated by three or more stages such as a percentage. For example, when the reference database is composed of a neural network, the first degree of association indicates a weight variable associated with the selected past evaluation target information.

<Generator 14>
The generation unit 14 generates an estimation result based on the evaluation information. The generation unit 14 uses format data such as an output format stored in advance in the storage unit 104 or the like to convert it into a format (for example, a character string) that can be understood by the user based on the information acquired as the evaluation result. Generate the converted information as an evaluation result.

Generator 14, for example, as shown in FIG. 2, to generate estimation result containing the string to be estimated such as "○○'s Hello.". The generation unit 14 may generate an estimation result including control information for controlling, for example, the estimation device 1 or the user terminal 3. For the setting of the format when generating the estimation result and the conversion method from the evaluation information to the estimation result, for example, a known technique (for example, artificial intelligence technique) may be used.

The generation unit 14 determines the content of the estimation result, for example, based on the first degree of association of the evaluation information. For example, the generation unit 14 generates an estimation result based on the reference information associated with the first association degree of "50%" or more, and uses the reference information associated with the first association degree of less than "50%" as the estimation result. It may be set not to reflect. As for the determination criteria based on the first degree of association, for example, the administrator or the like may set a threshold value or the like in advance, and the threshold range or the like can be arbitrarily set. Further, the generation unit 14 may determine the content of the estimation result based on, for example, the result of calculating two or more first association degrees and the comparison of two or more first association degrees.

The estimation result may include, for example, evaluation information. Moreover, the estimation result may include the target information.

<Output unit 15>
The output unit 15 outputs the estimation result. The output unit 15 transmits the estimation result to the output unit 109 via the I / F 107, and also transmits the estimation result to the user terminal 3 or the like via, for example, the I / F 105.

The output unit 15 may determine whether or not to transmit the estimation result to the user terminal 3, based on, for example, the estimation result. As the determination standard, for example, the administrator or the like may set a threshold value or the like in advance.

<Update part 16>
The update unit 16 updates, for example, the reference database. When the update unit 16 newly acquires the relationship between the past target information and the reference information, the update unit 16 reflects the relationship in the degree of association. For example, when the user evaluates the estimation accuracy of the keyboard input based on the estimation result generated by the generation unit 14 and the estimation device 1 acquires the evaluation result, the update unit 16 is included in the reference database based on the evaluation result. Update the degree of association.

<Server 2>
For example, the server 2 stores the above-mentioned various information. Various information sent via the communication network 4, for example, is stored in the server 2. For example, the server 2 may store the same information as the storage unit 104, and may send and receive various information to and from the estimation device 1 via the communication network 4. That is, the estimation device 1 may use the server 2 instead of the storage unit 104.

<User terminal 3>
The user terminal 3 has a projection unit that projects the virtual keyboard 8 described above, and a camera for generating target information. As the user terminal 3, a mobile phone (mobile terminal) is mainly used, and other than that, it is embodied in all electronic devices in addition to electronic devices such as smartphones, tablet terminals, wearable terminals, personal computers, and IoT devices. May be used. The user terminal 3 may be connected to the estimation device 1 via, for example, the communication network 4, or may be directly connected to, for example, the estimation device 1. The user acquires the estimation result from the estimation device 1 by using, for example, the user terminal 3. The user terminal 3 may include a projection unit and a plurality of cameras provided independently of each other.

<Communication network 4>
The communication network 4 is, for example, an Internet network or the like to which an estimation device 1 or the like is connected via a communication circuit. The communication network 4 may be composed of a so-called optical fiber communication network. Further, the communication network 4 may be realized by a known communication network such as a wireless communication network in addition to the wired communication network.

(Example of operation of estimation system 100)
Next, an example of the operation of the estimation system 100 in this embodiment will be described. FIG. 8 is a flowchart showing an example of the operation of the estimation system 100 in the present embodiment.

<Acquisition means S110>
As shown in FIG. 8, the target information including the angle information is acquired (acquisition means S110). The acquisition unit 11 acquires target information including angle information based on shooting information. The virtual keyboard 8 is projected by, for example, a projection unit of the user terminal 3 or the estimation device 1. The shooting information is generated by, for example, various cameras included in the user terminal 3 or the estimation device 1.

The acquisition unit 11 acquires the shooting information and acquires the angle information from the shooting information. As a result, the target information including the angle information is acquired. The acquisition unit 11 stores the target information in the storage unit 104, for example, via the storage unit 12. The acquisition unit 11 may acquire image data or the like after acquiring at least one of keyboard information and language information in advance.

The acquisition unit 11 may acquire the target information as the target information each time the shooting information is generated, or may acquire the target information generated in an arbitrary period, for example.

As shown in FIG. 3, for example, the acquisition means S110 includes a photographing information acquisition means S111, a coordinate data acquisition means S112, and an angle information acquisition means S113. In the shooting information acquisition means S111, the acquisition unit 11 acquires shooting information, and acquires, for example, three-dimensional image data generated by using a three-dimensional camera. In the coordinate data acquisition means S112, the acquisition unit 11 may acquire the coordinate data from the shooting information, or may acquire the coordinate data generated by, for example, the distance measurement camera. In the angle information acquisition means S113, the acquisition unit 11 acquires the angle information from the coordinate data. As the target for acquiring the angle information, an angle with the first joint of the finger as the apex, an angle with the second joint as the apex, and the like are used, and can be arbitrarily set according to the situation.

The acquisition means S110 includes, for example, an optical flow information acquisition means S114. In the optical flow information acquisition means S114, the acquisition unit 11 acquires an RGB image generated by using an RGB camera, and acquires optical flow information from a plurality of RGB images.

<Evaluation means S120>
Next, the reference database is referred to, and evaluation information including the first degree of association between the target information and the reference information is acquired (evaluation means S120). The evaluation unit 13 acquires the target information acquired by the acquisition unit 11, and acquires, for example, the reference database stored in the storage unit 104. The evaluation unit 13 may acquire one evaluation information for one target information, or may acquire one evaluation information for a plurality of target information, for example. The evaluation unit 13 stores the evaluation information in the storage unit 104, for example, via the storage unit 12.

<Generation means S130>
Next, an estimation result is generated based on the evaluation information (generation means S130). The generation unit 14 acquires the evaluation information acquired by the evaluation unit 13, and acquires format data such as an output format stored in the storage unit 104, for example. In addition to acquiring one estimation result for one evaluation information, the generation unit 14 may acquire one estimation result for a plurality of evaluation information, for example. The generation unit 14 stores the estimation result in the storage unit 104 via, for example, the storage unit 12.

As a result, for example, the output unit 15 transmits the estimation result to the user terminal 3 or the like or outputs the estimation result to the output unit 109, and the operation of the estimation system 100 in the present embodiment ends.

<Update means S140>
In addition, for example, when the relationship between the past target information and the reference information is newly acquired, the relationship may be reflected in the degree of association (update means S140). For example, when the user evaluates the estimation accuracy of the keyboard input based on the estimation result generated by the generation unit 14 and the estimation device 1 acquires the evaluation result, the update unit 16 is included in the reference database based on the evaluation result. Update the degree of association.

As a result, the operation of the estimation system 100 in the present embodiment may be completed. The timing when the update means S140 is implemented is arbitrary.

According to the present embodiment, the evaluation means S120 refers to the reference database and acquires the evaluation information including the first degree of association between the target information and the reference information in three or more stages. Therefore, it is possible to generate an estimation result based on the past results, and it is possible to estimate the key input in consideration of the variation in the pressing depth. This makes it possible to improve the accuracy of estimating the key input.

Further, according to the present embodiment, the target information includes angle information. Therefore, as compared with the case of calculating the position of the hand from one image, the positional relationship of the hand with respect to the virtual keyboard 8 can be easily acquired regardless of the characteristics of the hand that are different for each user. This makes it possible to acquire the information necessary for estimating the key input with high accuracy.

Further, according to the present embodiment, the coordinate data acquisition means S112 acquires the coordinate data from the shooting information. Further, the angle information acquisition means S113 acquires angle information from the coordinate data. Therefore, the amount of data to be evaluated can be reduced. As a result, when the target information is stored, it is possible to suppress an increase in the storage capacity.

Further, according to the present embodiment, the shooting information includes three-dimensional image data generated by using the three-dimensional camera. Therefore, when acquiring the angle information, a conventional three-dimensional camera can be used without providing a new camera or the like. This makes it possible to easily prepare an environment for estimating key input.

Further, according to the present embodiment, the target information and the past target information include optical flow information. Therefore, by combining the optical flow information and the angle information, it is possible to consider the movement at the time of key input for each user. This makes it possible to further improve the accuracy of estimating the key input.

Further, according to the present embodiment, the acquisition means S110 acquires optical flow information from a plurality of RGB images. Therefore, when acquiring optical flow information, a conventional RGB camera can be used without providing a new camera or the like. This makes it possible to easily prepare an environment for improving the estimation accuracy of key input.

Further, according to the present embodiment, the target information and the past target information include keyboard information. Therefore, even when a virtual keyboard 8 having a different key layout or the like is used, it is possible to estimate the key input with high accuracy.

Further, according to the present embodiment, the reference database is constructed by machine learning. Therefore, it is possible to estimate the key input excluding the subjectivity as compared with the case of setting the threshold value for the pressing depth calculated from the image.

According to the present embodiment, the evaluation unit 13 refers to the reference database and acquires the evaluation information including the first degree of association between the target information and the reference information in three or more stages. Therefore, it is possible to generate an estimation result based on the past results, and it is possible to estimate the key input in consideration of the variation in the pressing depth. This makes it possible to improve the accuracy of estimating the key input.

Further, according to the present embodiment, the target information includes angle information. Therefore, as compared with the case of calculating the position of the hand from one image, the positional relationship of the hand with respect to the virtual keyboard 8 can be easily acquired regardless of the characteristics of the hand that are different for each user. This makes it possible to acquire the information necessary for estimating the key input with high accuracy.

According to the present embodiment, the target information and the past target information may include linguistic information. In this case, the key input can be estimated with high accuracy without depending on the language used by the user.

According to the present embodiment, when the updating means S140 newly acquires the relationship between the past target information and the reference information, the relationship may be reflected in the degree of association. In this case, the degree of association can be easily updated, and the continuous estimation accuracy can be improved.

According to the present embodiment, the image data may include RGB image data and infrared image data. In this case, even when the imaging environment is different due to lighting or the like, the shape of the hand or the like can be acquired with high accuracy. This makes it possible to further improve the accuracy of estimating the key input.

In addition, according to this embodiment, the position information may have distance data. In this case, the position of the hand with respect to the depth direction can be easily obtained. This makes it possible to further improve the accuracy of estimating the key input.

According to the present embodiment, the projection unit, the RGB camera, the infrared camera, and the distance measurement camera may be provided in the same terminal. In this case, the virtual keyboard 8 can be used regardless of the location.

Although embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1: Estimator 2: Server 3: User terminal 4: Communication network 8: Virtual keyboard 10: Housing 11: Acquisition unit 12: Storage unit 13: Evaluation unit 14: Generation unit 15: Output unit 16: Update unit 100: Estimate System 101: CPU
102: ROM
103: RAM
104: Preservation unit 105: I / F
106: I / F
107: I / F
108: Input part 109: Output part 110: Internal bus S110: Acquisition means S120: Evaluation means S130: Generation means S140: Update means

The estimation system according to the first invention is an estimation system that estimates a character string input by using a virtual keyboard, and is based on shooting information obtained by photographing an action of the user pressing the virtual keyboard, and is based on shooting information of the user's finger. Three or more levels of association between the acquisition means for acquiring the target information including the angle information regarding the angle, the past target information acquired in advance, and the reference information including the character string input by the past target information. An evaluation that refers to the reference database in which is stored and the reference database, and acquires evaluation information including a first degree of association between the target information acquired by the acquisition means and the reference information in three or more stages. A means and a generation means for generating an estimation result based on the evaluation information are provided , and the angle of the finger is calculated from the coordinates of three or more points centered on the coordinates at the joint portion of the coordinates of the user's finger. It is characterized by showing an angle .

The estimation system according to the fifth invention is characterized in that, in any one of the first to fourth inventions, the target information and the past target information include keyboard information indicating the type of the virtual keyboard.

The estimation system according to the sixth invention is characterized in that, in any one of the first to fifth inventions, the reference database is constructed by machine learning.

The estimation device according to the seventh invention is an estimation device that estimates a character string input by using a virtual keyboard, and is based on shooting information obtained by photographing an action of the user pressing the virtual keyboard, and is a finger of the user. Three or more levels of association between the acquisition unit that acquires the target information including the angle information related to the angle, the past target information acquired in advance, and the reference information including the character string input by the past target information. An evaluation that refers to the reference database in which is stored and the reference database, and acquires evaluation information including a first degree of association between the target information acquired by the acquisition unit and the reference information in three or more stages. A unit and a generation unit that generates an estimation result based on the evaluation information are provided , and the angle of the finger is calculated from three or more points of the coordinates of the user's finger centered on the coordinates at the joint portion. It is characterized by showing an angle .

According to the first to sixth inventions, the evaluation means refers to the reference database and acquires the evaluation information including the first degree of association between the target information and the reference information in three or more stages. Therefore, it is possible to generate an estimation result based on the past results, and it is possible to estimate the key input in consideration of the variation in the pressing depth. This makes it possible to improve the accuracy of estimating the key input.

Further, according to the first to sixth inventions, the target information includes the angle information. Therefore, as compared with the case of calculating the position of the hand from one image, the positional relationship of the hand with respect to the virtual keyboard can be easily acquired regardless of the characteristics of the hand that are different for each user. This makes it possible to acquire the information necessary for estimating the key input with high accuracy.

Further, according to the fifth invention, the target information and the past target information include keyboard information. Therefore, even when a virtual keyboard having a different key layout or the like is used, it is possible to estimate the key input with high accuracy.

Further, according to the sixth invention, the reference database is constructed by machine learning. Therefore, it is possible to estimate the key input excluding the subjectivity as compared with the case of setting the threshold value for the pressing depth calculated from the image.

According to the seventh invention, the evaluation unit refers to the reference database and acquires the evaluation information including the first degree of association between the target information and the reference information in three or more stages. Therefore, it is possible to generate an estimation result based on the past results, and it is possible to estimate the key input in consideration of the variation in the pressing depth. This makes it possible to improve the accuracy of estimating the key input.

Further, according to the seventh invention, the target information includes angle information. Therefore, as compared with the case of calculating the position of the hand from one image, the positional relationship of the hand with respect to the virtual keyboard can be easily acquired regardless of the characteristics of the hand that are different for each user. This makes it possible to acquire the information necessary for estimating the key input with high accuracy.

Claims (8)

An estimation system that estimates the character string entered using a virtual keyboard.
An acquisition means for acquiring target information including angle information regarding the angle of the user's finger based on shooting information obtained by photographing the operation of the user pressing the virtual keyboard.
A reference database that stores three or more levels of association between the past target information acquired in advance and the reference information including the character string input by the past target information.
An evaluation means that refers to the reference database and acquires evaluation information including the first degree of association between the target information and the reference information in three or more stages.
A generation means for generating an estimation result based on the evaluation information, and
An estimation system characterized by being equipped with. The acquisition means
A shooting information acquisition means for acquiring the shooting information and
A coordinate data acquisition means for acquiring the coordinate data of the user's finger from the shooting information, and
An angle information acquisition means for acquiring the angle information from the coordinate data,
The estimation system according to claim 1, wherein the estimation system is characterized by having. The shooting information includes three-dimensional image data generated by using a three-dimensional camera.
The estimation system according to claim 2, wherein the coordinate data acquisition means acquires the coordinate data from the three-dimensional image data. The estimation system according to any one of claims 1 to 3, wherein the target information and the past target information include optical flow information indicating the movement of the user's finger. The shooting information includes an RGB image generated by using an RGB camera.
The estimation system according to claim 4, wherein the acquisition means includes an optical flow information acquisition means for acquiring the optical flow information from a plurality of the RGB images. The estimation system according to any one of claims 1 to 5, wherein the target information and the past target information include keyboard information indicating the type of the virtual keyboard. The estimation system according to any one of claims 1 to 6, wherein the reference database is constructed by machine learning. An estimation device that estimates a character string input using a virtual keyboard.
An acquisition unit that acquires target information including angle information regarding the angle of the user's finger based on shooting information obtained by photographing the operation of the user pressing the virtual keyboard.
A reference database that stores three or more levels of association between the past target information acquired in advance and the reference information including the character string input by the past target information.
An evaluation unit that refers to the reference database and acquires evaluation information including the first degree of association between the target information and the reference information in three or more stages.
A generator that generates an estimation result based on the evaluation information,
An estimation device characterized by comprising.

Images