JP6510134B1 - Estimation system and estimation apparatus - Google Patents

Abstract

An estimation system and estimation apparatus capable of improving the accuracy of estimating a key input are provided. An estimation system for estimating a character string input using a virtual keyboard, comprising: moving image data obtained by photographing an operation of the user pressing the virtual keyboard; and position information of a user's hand in the moving image data Between acquisition means S110 for acquiring target information to be included, past target information acquired in advance, reference information including a character string inputted by the target information in the past, and between the target information in the past and the reference information Evaluation that acquires evaluation information including three or more first association degrees between the target information and the reference information with reference to the reference database in which three or more stages of association degrees are stored and the reference database It is characterized by comprising: means S120; and generation means S130 for generating an estimation result based on the evaluation information. [Selected figure] Figure 1

Description

  The present invention relates to an estimation system that estimates a character string input using a virtual keyboard, and an estimation apparatus.

  Conventionally, as a technique for controlling key input without using a physical keyboard, for example, an information processing apparatus of Patent Document 1 has been proposed.

  In Patent Document 1, an extraction unit that extracts a hand area from an image input by an imaging unit that captures a hand, a reference line determination unit that determines a predetermined line in the image as a pressed reference line based on the hand area, A first position determination unit that determines the position in the depth direction based on a determination unit that determines that the lower part of the area exceeds the press reference line and that is determined to be pressed and an aspect ratio of the hand area that is determined to be pressed by the determination unit And the second position determination unit that determines the horizontal position based on the position of the lowermost part of the hand area when the determination unit determines that the button is pressed, and the input is performed based on the determined position in the depth direction and the horizontal position An information processing apparatus is disclosed that includes an input key determination unit that determines a key.

International Publication No. 2012/176315

  Here, for example, in an information processing apparatus such as Patent Document 1, an input command is estimated based on a threshold value based on an image captured from one direction by one camera. For this reason, it is necessary to perform key input according to the threshold value, and there is a concern that variation in depression depth can not be considered, and it is difficult to improve estimation accuracy of key input. Therefore, the improvement of the accuracy of estimating the key input is mentioned as a problem.

  Therefore, the present invention has been made in view of the above-described problems, and an object thereof is to provide an estimation system and estimation apparatus capable of improving the accuracy of estimating key input. It is in.

  An estimation system according to a first aspect of the present invention is an estimation system for estimating a character string input using a virtual keyboard, comprising: moving image data obtained by photographing an operation of the user pressing the virtual keyboard; Acquisition means for acquiring target information including hand position information, past target information acquired in advance, reference information including a character string input by the past target information, and the past target information and the reference Evaluation information including a reference database in which three or more levels of association with information are stored, and the reference database with reference to the target information and three or more levels of first association between the reference information And generating means for generating an estimation result based on the evaluation information.

  An estimation system according to a second aspect of the present invention is the estimation system according to the first aspect, wherein the moving image data uses RGB moving image data generated using an RGB camera corresponding to a visible light region and an infrared camera corresponding to an infrared light region. And generating infrared moving image data.

  In the estimation system according to the third aspect of the present invention, in the second aspect, the position information is generated using a distance measurement camera that measures a distance from an object, and between the distance measurement camera and the user's hand It has the distance data which shows the distance of.

  The estimation system according to a fourth aspect is characterized in that, in the third aspect, the projection unit for projecting the virtual keyboard, the RGB camera, the infrared camera, and the distance measurement camera are provided in the same terminal.

  The estimation system according to the fifth invention is characterized in that, in any one of the first invention to the fourth invention, the target information and the past target information include keyboard information indicating a type of the virtual keyboard.

  The estimation system according to the sixth invention is characterized in that, in any one of the first invention to the fifth invention, the target information and the past target information include linguistic information.

  The estimation system according to the seventh invention is characterized in that, in any one of the first invention to the sixth invention, the reference database is constructed by machine learning.

  The estimation system according to an eighth aspect of the present invention is the estimation system according to any one of the first to seventh aspects, in the case where a relation between the past target information and the reference information is newly acquired, the relation is the relation The apparatus is characterized by further comprising update means for reflecting on each occasion.

  An estimation apparatus according to a ninth aspect of the present invention is an estimation apparatus for estimating a character string input using a virtual keyboard, comprising: moving image data obtained by photographing an operation of the user pressing the virtual keyboard; Acquisition unit for acquiring target information including hand position information, past target information acquired in advance, reference information including a character string input by the past target information, and the past target information and the reference Evaluation information including a reference database in which three or more levels of association with information are stored, and the reference database with reference to the target information and three or more levels of first association between the reference information And a generation unit that generates an estimation result based on the evaluation information.

  According to the first invention to the eighth invention, the evaluation means refers to the reference database and acquires evaluation information including the first degree of association between the target information and the reference information in three or more stages. For this reason, the estimation result based on the past result can be generated, and the key input can be estimated in consideration of the variation in depression depth. This makes it possible to improve the accuracy of estimating the key input.

  Further, according to the first to eighth inventions, the target information includes moving image data and position information. For this reason, compared with the case of calculating the position of the hand from one image, the detailed positional relationship of the hand with respect to the virtual keyboard can be acquired. This makes it possible to obtain the information necessary to estimate the key input with high accuracy.

  Further, according to the second invention, the moving image data includes RGB moving image data and infrared moving image data. For this reason, even when the imaging environment resulting from the illumination or the like is different, the shape or the like of the hand can be acquired with high accuracy. This makes it possible to further improve the accuracy of estimating key input.

  Further, according to the third invention, the position information has distance data. Therefore, the position of the hand in the depth direction can be easily obtained. This makes it possible to further improve the accuracy of estimating key input.

  According to the fourth aspect of the invention, the projection unit, the RGB camera, the infrared camera, and the distance measurement camera are mounted on the same terminal. Therefore, it is possible to use the virtual keyboard regardless of the place.

  Further, according to the fifth invention, the target information and the past target information include keyboard information. Therefore, even when virtual keyboards with different key arrangements and the like are used, it is possible to estimate key input with high accuracy.

  Further, according to the sixth invention, the target information and the past target information include language information. Therefore, key input can be estimated with high accuracy without depending on the language used by the user.

  Also, according to the seventh invention, the reference database is constructed by machine learning. For this reason, it is possible to realize estimation of key input excluding subjectivity, as compared with the case of setting a threshold for the pressing depth calculated from the image.

  Further, according to the eighth aspect of the invention, the updating means reflects the relationship in the degree of association when newly acquiring the relationship between the past target information and the reference information. For this reason, it is possible to easily update the degree of association and to improve the estimation accuracy continuously.

  According to the ninth aspect, the evaluation unit refers to the reference database, and acquires evaluation information including the first degree of association between the target information and the reference information in three or more stages. For this reason, the estimation result based on the past result can be generated, and the key input can be estimated in consideration of the variation in depression depth. This makes it possible to improve the accuracy of estimating the key input.

  Further, according to the ninth aspect, the target information includes moving image data and position information. For this reason, compared with the case of calculating the position of the hand from one image, the detailed positional relationship of the hand with respect to the virtual keyboard can be acquired. This makes it possible to obtain the information necessary to estimate the key input with high accuracy.

Fig.1 (a) is a schematic diagram which shows an example of the estimation system in this embodiment, FIG.1 (b) is a schematic diagram which shows the other example of the estimation system in this embodiment. FIG. 2 is a schematic view showing an example of the operation of the estimation system. Fig.3 (a) is a schematic diagram which shows an example of a structure of the estimation apparatus in this embodiment, FIG.3 (b) is a schematic diagram which shows an example of a function of the estimation apparatus in this embodiment. FIG. 4 is a schematic view showing an example of the reference database in the present embodiment. FIG. 5 is a schematic view showing a first modified example of the reference database in the present embodiment. FIG. 6 is a schematic view showing a second modification of the reference database in the present embodiment. FIG. 7 is a flowchart showing an example of the operation of the estimation system in the present embodiment.

  Hereinafter, an example of an estimation system and an estimation apparatus in an embodiment to which the present invention is applied will be described with reference to the drawings.

  An example of the estimation system 100 and the estimation device 1 in the present embodiment will be described with reference to FIG. 1.

  The estimation system 100 in the present embodiment includes an estimation device 1 as shown in FIG. 1A, for example. The estimation device 1 is connected to the user terminal 3 via, for example, the communication network 4. The estimation system 100 estimates a character string input using the virtual keyboard 8. In the estimation system 100, in addition to the virtual keyboard 8 being projected by the user terminal 3, the virtual keyboard 8 may be projected by the estimation device 1 as shown in FIG. 1 (b), for example. Eight may be projected.

  In the estimation system 100, for example, as shown in FIG. 2, the estimation device 1 acquires target information and generates an estimation result for the target information. The target information includes moving image data obtained by capturing an operation of the user pressing the virtual keyboard 8 and positional information of the user's hand in the moving image data. The target information is generated not only by the user terminal 3 but also by the estimation device 1, for example.

  The moving image data may be generated using a known camera provided in the user terminal 3 or the estimation device 1, or may be generated using a known camera independent of the user terminal 3 or the estimation device 1. The moving image data indicates a moving image obtained by capturing the movement of the user's hand before and after pressing at least one key. The moving image data may be, for example, a moving image obtained by imaging at least the movement of the user's fingertip, in addition to the moving image obtained by imaging the movement from the user's wrist to the fingertip. The moving image data includes, for example, information on the keyboard such as a shape of the virtual keyboard 8, a projection size, a key height, and a key arrangement (equivalent to keyboard information described later).

  The moving image data includes, for example, RGB moving image data and infrared moving image data. RGB moving image data is generated using an RGB camera corresponding to a visible light region (for example, 400 nm or more and 800 nm or less). Infrared moving image data is generated using an infrared camera corresponding to an infrared light region (for example, 800 nm or more and 1,000 μm or less). RGB moving image data and infrared moving image data may be generated using, for example, one multispectral camera corresponding to the visible light region to the infrared light region.

  The position information is generated using a known distance measurement camera that measures the distance from the object. The position information is generated using a distance measurement camera provided in the user terminal 3 or the estimation device 1, and may be generated using a distance measurement camera independent of the user terminal 3 or the estimation device 1.

  The position information has distance data indicating the distance between the distance measurement camera and the user's hand (object) captured by the moving image data. The distance data indicates the distance between the distance measurement camera and at least one place of the user's hand set in advance. The distance data may indicate, for example, the distance between the distance measurement camera and each finger of the user. The user's fingertip indicates a portion assumed to be in contact with the virtual keyboard 8. For example, a total of ten locations corresponding to the fingertip of the right hand and the fingertip of the left hand are targets of distance data.

  The distance data is generated, for example, in the same period as shooting of moving image data. The distance data may be generated, for example, only during a period before and after the user presses the virtual keyboard 8. The distance data is generated as numerical data, and may be indicated by a vector or a matrix.

  The position information may include, for example, coordinate data. The coordinate data is represented by, for example, a coordinate space using x-axis, y-axis and z-axis based on the position of the distance measurement camera. Similar to distance information, the position information may indicate coordinates of at least one place of the user's hand, for example, coordinates of each finger of the user. The coordinate data is generated, for example, in the same period as shooting of moving image data. The coordinate data may be generated, for example, only during a period before and after the user presses the virtual keyboard 8. Coordinate data is generated as numerical data, and may be indicated by a vector or a matrix.

  In addition, the projection part which projects the virtual keyboard 8 mentioned above, an RGB camera, an infrared camera, and a distance measurement camera may be equipped with the same terminal (for example, the estimation apparatus 1 or the user terminal 3). Also, the above-described moving image data and position information may be generated using a known camera such as Kinect (registered trademark) or a multispectral camera, and in this case, for example, a camera with two or more eyes may be used.

  The target information may include, for example, keyboard information. The keyboard information indicates the type of virtual keyboard 8 used by the user. The keyboard information includes, for example, the arrangement of keys, the size of each key, the key height and the like in addition to the projection size of the virtual keyboard 8.

  The target information may include, for example, language information. The language information indicates the type of language input by the user using the virtual keyboard 8. In the estimation system 100, the input character string may be estimated based on the language information. In this case, it becomes possible to estimate key input with high accuracy without depending on the language used by the user. In addition, it is possible to generate an estimation result in consideration of a typo or the like.

  The estimation result indicates the result of estimation of the content corresponding to the input key when the user presses the virtual keyboard 8. As a result of estimation, a character string is generated, and in addition, for example, a control command of the estimation device 1 or the user terminal 3 is generated. In the estimation system 100, the accuracy of the estimation result can be improved by increasing the types of information included in the target information. In particular, the combination of the moving image data and the position information makes it possible to capture the movement of the user's finger with high accuracy.

(Estimation device 1)
Next, referring to FIG. 3, an example of the estimation device 1 in the present embodiment will be described. Fig.3 (a) is a schematic diagram which shows an example of a structure of the estimation apparatus 1 in this embodiment, FIG.3 (b) is a schematic diagram which shows an example of a function of the estimation apparatus 1 in this embodiment.

  For example, an electronic device such as a personal computer (PC) is used as the estimation device 1. For example, a smartphone, a tablet terminal, a wearable terminal, an electronic device such as an IoT (Internet of Things) device, Raspberry Pi (registered trademark), etc. A single board computer may be used. For example, as illustrated in FIG. 3A, the estimation apparatus 1 includes a housing 10, a central processing unit (CPU) 101, a read only memory (ROM) 102, a random access memory (RAM) 103, and a storage unit. 104 and I / Fs 105-107. Each configuration 101-107 is connected by an internal bus 110.

  The CPU 101 controls the entire estimation device 1. The ROM 102 stores an operation code of the CPU 101. The RAM 103 is a work area used when the CPU 101 operates. The storage unit 104 stores various information such as sound data. As the storage unit 104, for example, a data storage device such as a solid state drive (SSD) or a floppy disk may be used in addition to a hard disk drive (HDD). For example, the estimation device 1 may have a GPU (Graphics Processing Unit) not shown. By having a GPU, higher-speed arithmetic processing can be performed than usual.

  The I / F 105 is an interface for transmitting and receiving various information to and from the user terminal 3 and the server 2 via the communication network 4 such as the Internet. The I / F 106 is an interface for transmitting and receiving information with the input unit 108. For example, a keyboard is used as the input unit 108, and a manager or the like of the estimation apparatus 1 inputs various information or control commands of the estimation apparatus 1 via the input unit 108. The I / F 107 is an interface for transmitting and receiving various information to and from the output unit 109. The output unit 109 outputs various types of information stored in the storage unit 104, the processing status of the estimation apparatus 1, and the like. A display is used as the output portion 109, and may be, for example, a touch panel.

  For example, when the output unit 109 includes a projection unit that projects the virtual keyboard 8, the input unit 108 includes various cameras (for example, the above-described RGB camera, infrared camera, and distance measurement camera) for generating target information. You may have. When a terminal such as a smartphone or a tablet is used as the estimation device 1, a known known projection device (projection unit) provided is used as a part of the output unit 109, and a known camera provided (imaging unit) May be used as part of the input portion 108.

<Reference database>
The storage unit 104 stores a reference database in which past target information, reference information, and the degree of association acquired in advance are stored. The past target information indicates the same type of information as the above target information, includes past image data and past position information, and may include, for example, past keyboard information. The reference information includes a character string input by past target information, and includes, for example, a control command of the estimation device 1 or the user terminal 3. The degree of association indicates the degree of relationship between past target information and reference information.

  For example, as shown in FIG. 4, the reference database stores three or more levels of association between past target information and reference information. The degree of association is indicated by, for example, three or more steps such as percentage, 10 steps, or 5 steps, and is indicated by, for example, line characteristics (eg, thickness etc.). The reference database is formed by an algorithm that can calculate the degree of association using, for example, past target information and reference information. The past target information and reference information have a plurality of data, and the relation between each past target information and each reference information is linked by the degree of association.

  For example, "information A" included in the past target information indicates the degree of association "80%" with "reference A" included in the reference information, and between "reference B" included in the reference information Indicates the degree of association “15%”. That is, the “degree of association” indicates the degree of connection between each piece of data, and the higher the degree of association, the stronger the connection between each piece of data.

  In addition, past target information is stored in the reference database in the form of video data (a plurality of image data indicating temporal change) or a character string etc. For example, the form such as numerical value, matrix (vector) or histogram May be stored. The reference information is stored in the reference database in the form of a character string or the like, and may be stored, for example, in the form of a numerical value, a matrix (vector), or a histogram. That is, “information A”, “information B”, and “information C” shown in FIG. 4 are the target information (eg, RGB moving image data, infrared moving image data, distance data, coordinate data, and keyboard information) acquired in the past. At least one of them is data stored in the form of video data, a matrix, etc., and “reference A”, “reference B” and “reference C” are characters corresponding to estimation results obtained in the past Indicates data stored in the form of a column, a matrix, or the like.

  The reference database is constructed using, for example, machine learning. In this case, the degree of association is calculated using machine learning. For example, deep learning is used for machine learning. The reference database is constructed, for example, by a neural network, in which case the degree of association may be indicated by a hidden layer and a weight variable. That is, in the estimation system 100, a reference database constructed by machine learning can be used. In this case, a complex relationship between past target information having a plurality of data and the reference information is associated with each other. Can be shown with high accuracy. This makes it possible to improve the accuracy of the estimation result.

  For example, as shown in FIG. 5, past target information may be stored in a reference database by dividing past moving image data and past position information. In this case, the degree of association is calculated based on the relationship between the combination of the past moving image data and the past position information and the reference information.

  For example, the combination of “moving image A” included in past moving image data and “position A” included in past position information indicates the degree of association “90%” with “reference A”, “reference B” Indicates a degree of association of “20%”. In this case, data can be stored independently of past moving image data and past position information. Therefore, when generating the estimation result, it is possible to improve the accuracy and expand the range of options.

  The past moving image data may be stored, for example, in the reference database, by dividing the past RGB moving image data and the past infrared moving image data. In addition, past position information may be stored, for example, in the reference database by dividing the past distance data and the past coordinate data.

  For example, as shown in FIG. 6, past target information may include synthetic data and a degree of similarity. The synthetic data is indicated by three or more stages of similarity with past moving image data or past position information. The synthetic data is stored in the reference database in the form of video data, a character string or the like, and may be stored, for example, in the form of a numerical value, a matrix, or a histogram.

  FIG. 3 (b) is a schematic view showing an example of the function of the estimation device 1. The estimation apparatus 1 includes an acquisition unit 11, a storage unit 12, an evaluation unit 13, a generation unit 14, and an output unit 15, and may include, for example, an update unit 16. Each function illustrated in FIG. 3B may be realized by the CPU 101 executing a program stored in the storage unit 104 or the like using the RAM 103 as a work area, and may be controlled by artificial intelligence, for example.

<Acquisition unit 11>
The acquisition unit 11 acquires target information. The acquisition unit 11 acquires target information from an external terminal such as the user terminal 3 via the communication network 4 and the I / F 105, and also, for example, an imaging unit of the input unit 108 (for example, the above-described RGB camera, infrared camera, and Object information is acquired from at least one of the distance measurement cameras. The acquiring unit 11 may acquire moving image data and position information separately as object information and may acquire moving image data and position information collectively as object information, for example. The frequency and period at which the acquisition unit 11 acquires target information are arbitrary.

  The acquisition unit 11 receives various information transmitted to the estimation device 1. The acquisition unit 11 receives various information such as target information transmitted from an external terminal such as the user terminal 3 via the communication network 4 and the I / F 105.

<Storage unit 12>
The storage unit 12 extracts various information such as a reference database stored in the storage unit 104 as needed. The storage unit 12 stores, in the storage unit 104, various types of information acquired or generated by the configurations 11, 13-16.

<Evaluation part 13>
The evaluation unit 13 refers to the reference database and acquires evaluation information including the first degree of association between the target information and the reference information. For example, when referring to the reference database shown in FIG. 4, the evaluation unit 13 refers to past target information (for example, “information A”: first target information in the past) that is the same as or similar to information included in the target information. select. As target information in the past, information that partially or completely matches the target information is selected, and, for example, information that is similar (including the same concept or the like) is selected. When the target information is represented by a numerical value such as a matrix, a numerical range included in the selected past target information may be set in advance.

  The evaluation unit 13 selects reference information linked to the selected past first target information, and a degree of association (first degree of association) between the selected past target information and the reference information, and acquires it as evaluation information. . The first association degree may be calculated by the evaluation unit 13 in addition to the selection of a part of the association degree.

  For example, the evaluation unit 13 selects the reference information “reference A” linked to the first target information “information A”, and the first association degree “80%” between the “information A” and the “reference A”, Acquire as evaluation information. The reference information and the first association degree may include a plurality of data. In this case, in addition to the “reference A” and “80%” described above, reference information “reference B” linked to the first target information “information A”, and “information A” and “reference B” The first association degree “15%” may be selected, and “reference A” and “80%”, and “reference B” and “15%” may be acquired as evaluation information.

  The evaluation information may include target information. Also, the first association degree is indicated by three or more stages such as percentage. For example, when the reference database is configured by a neural network, the first association degree indicates a weight variable linked to the selected past evaluation object information.

<Generation unit 14>
The generation unit 14 generates an estimation result based on the evaluation information. The generation unit 14 converts, for example, a format (for example, a character string) that can be understood by the user based on the information acquired as the evaluation result, using format data such as an output format stored in advance in the storage unit 104 or the like. The converted information is generated as an evaluation result.

  Generator 14, for example, as shown in FIG. 2, to generate estimation result containing the string to be estimated such as "○○'s Hello.". The generation unit 14 may generate, for example, an estimation result including control information for controlling the estimation device 1 or the user terminal 3. Note that, for example, a known technique (for example, an artificial intelligence technique) may be used as the setting of the format when generating the estimation result and the conversion method from the evaluation information to the estimation result.

  The generation unit 14 determines the content of the estimation result based on, for example, the first degree of association of the evaluation information. For example, the generation unit 14 generates an estimation result based on the reference information associated with the first association degree of “50%” or more, and generates the reference information associated with the first association degree less than “50%” as the estimation result. It may be set not to reflect. Note that, for example, a manager or the like may set a threshold or the like in advance, for example, as a determination criterion based on the first association degree, and the range or the like of the threshold can be set arbitrarily. Also, the generation unit 14 may determine the content of the estimation result based on, for example, the result of computing two or more first association degrees, or the comparison of two or more first association degrees.

  The estimation result may include, for example, evaluation information. In addition, the estimation result may include target information.

<Output unit 15>
The output unit 15 outputs the estimation result. The output unit 15 transmits the estimation result to the output portion 109 via the I / F 107, and also transmits the estimation result to the user terminal 3 or the like via the I / F 105, for example.

  The output unit 15 may determine whether to transmit the estimation result to the user terminal 3 based on, for example, the estimation result. For example, a manager or the like may set a threshold or the like in advance as a determination criterion.

<Update unit 16>
The update unit 16 updates, for example, the reference database. The updating unit 16 reflects the relationship in the degree of association when newly acquiring a relationship between past target information and reference information. For example, when the user evaluates the estimation accuracy of the keyboard input based on the estimation result generated by the generation unit 14 and the estimation device 1 acquires the evaluation result, the updating unit 16 is included in the reference database based on the evaluation result. Update the degree of association.

<Server 2>
The server 2 stores, for example, the various information described above. For example, various information sent via the communication network 4 is accumulated in the server 2. For example, information similar to that of the storage unit 104 may be stored in the server 2, and transmission / reception of various information with the estimation apparatus 1 may be performed via the communication network 4. That is, the estimation device 1 may use the server 2 instead of the storage unit 104.

<User terminal 3>
The user terminal 3 has a projection unit that projects the virtual keyboard 8 described above, and a camera for generating target information. A mobile phone (portable terminal) is mainly used as the user terminal 3, and in addition to electronic devices such as a smart phone, a tablet type terminal, a wearable terminal, a personal computer, an IoT device, and the like, other electronic devices May be used. The user terminal 3 may be connected to, for example, the estimation device 1 via the communication network 4 or directly to, for example, the estimation device 1. The user acquires an estimation result from the estimation device 1 using, for example, the user terminal 3.

<Communication network 4>
The communication network 4 is, for example, an Internet network or the like to which the estimation device 1 or the like is connected via a communication circuit. The communication network 4 may be configured by a so-called optical fiber communication network. In addition to the wired communication network, the communication network 4 may be realized by a known communication network such as a wireless communication network.

(An example of operation of estimation system 100)
Next, an example of the operation of the estimation system 100 in the present embodiment will be described. FIG. 7 is a flowchart showing an example of the operation of the estimation system 100 in the present embodiment.

<Acquisition means S110>
As shown in FIG. 7, target information including moving image data and position information is acquired (acquisition means S110). The acquisition unit 11 acquires target information including moving image data obtained by capturing an operation in which the user presses the virtual keyboard 8 and position information of the user's hand in the moving image data. The virtual keyboard 8 is projected by the projection unit of the user terminal 3 or the estimation device 1, for example. The moving image data and the position information are generated by, for example, various cameras of the user terminal 3 or the estimation device 1. The acquisition unit 11 stores target information in the storage unit 104 via, for example, the storage unit 12. The acquiring unit 11 may acquire moving image data and the like after acquiring in advance at least one of keyboard information and language information.

  The acquisition unit 11 may acquire target information generated in an arbitrary period, for example, in addition to acquiring the target information each time at least one of moving image data and position information is generated.

<Evaluation means S120>
Next, with reference to the reference database, evaluation information including the first degree of association between the target information and the reference information is acquired (evaluation means S120). The evaluation unit 13 acquires the target information acquired by the acquisition unit 11, and acquires, for example, the reference database stored in the storage unit 104. The evaluation unit 13 may obtain one evaluation information for one target information, or may obtain one evaluation information for a plurality of target information, for example. The evaluation unit 13 stores the evaluation information in the storage unit 104 via, for example, the storage unit 12.

<Generation means S130>
Next, an estimation result is generated based on the evaluation information (generation means S130). The generation unit 14 acquires the evaluation information acquired by the evaluation unit 13, and acquires, for example, format data such as an output format stored in the storage unit 104. The generation unit 14 may obtain one estimation result for one evaluation information, or may obtain one estimation result for a plurality of evaluation information, for example. The generation unit 14 stores the estimation result in the storage unit 104 via, for example, the storage unit 12.

  Thereby, for example, the output unit 15 transmits the estimation result to the user terminal 3 or the like, or outputs the estimation result to the output unit 109, and the operation of the estimation system 100 in the present embodiment ends.

<Updating means S140>
In addition, for example, when the relationship between the past target information and the reference information is newly acquired, the relationship may be reflected on the degree of association (update unit S140). For example, when the user evaluates the estimation accuracy of the keyboard input based on the estimation result generated by the generation unit 14 and the estimation device 1 acquires the evaluation result, the updating unit 16 is included in the reference database based on the evaluation result. Update the degree of association.

  Thereby, the operation of the estimation system 100 in the present embodiment may end. In addition, the timing in the case of implementing update means S140 is arbitrary.

  According to the present embodiment, the evaluation unit S120 refers to the reference database, and acquires evaluation information including the first degree of association between the target information and the reference information in three or more stages. For this reason, the estimation result based on the past result can be generated, and the key input can be estimated in consideration of the variation in depression depth. This makes it possible to improve the accuracy of estimating the key input.

  Further, according to the present embodiment, the target information includes moving image data and position information. For this reason, compared with the case where the position of the hand is calculated from one image, the detailed positional relationship of the hand with respect to the virtual keyboard 8 can be acquired. This makes it possible to obtain the information necessary to estimate the key input with high accuracy.

  Further, according to the present embodiment, the moving image data includes RGB moving image data and infrared moving image data. For this reason, even when the imaging environment resulting from the illumination or the like is different, the shape or the like of the hand can be acquired with high accuracy. This makes it possible to further improve the accuracy of estimating key input.

  Further, according to the present embodiment, the position information has distance data. Therefore, the position of the hand in the depth direction can be easily obtained. This makes it possible to further improve the accuracy of estimating key input.

  Further, according to the present embodiment, the projection unit, the RGB camera, the infrared camera, and the distance measurement camera are provided in the same terminal. For this reason, it becomes possible to use the virtual keyboard 8 without choosing a place.

  Further, according to the present embodiment, the target information and the past target information include keyboard information. Therefore, even when the virtual keyboard 8 having a different key arrangement or the like is used, it is possible to estimate key input with high accuracy.

  Further, according to the present embodiment, the target information and the past target information include language information. Therefore, key input can be estimated with high accuracy without depending on the language used by the user.

  Further, according to the present embodiment, the reference database is constructed by machine learning. For this reason, it is possible to realize estimation of key input excluding subjectivity, as compared with the case of setting a threshold for the pressing depth calculated from the image.

  Further, according to the present embodiment, the updating unit S140 reflects the relationship in the degree of association when newly acquiring the relationship between the past target information and the reference information. For this reason, it is possible to easily update the degree of association and to improve the estimation accuracy continuously.

  According to the present embodiment, the evaluation unit 13 refers to the reference database, and acquires evaluation information including the first degree of association between the target information and the reference information in three or more stages. For this reason, the estimation result based on the past result can be generated, and the key input can be estimated in consideration of the variation in depression depth. This makes it possible to improve the accuracy of estimating the key input.

  Further, according to the present embodiment, the target information includes moving image data and position information. For this reason, compared with the case where the position of the hand is calculated from one image, the detailed positional relationship of the hand with respect to the virtual keyboard 8 can be acquired. This makes it possible to obtain the information necessary to estimate the key input with high accuracy.

  While the embodiments of the present invention have been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and the gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

1: estimation device 2: server 3: user terminal 4: communication network 8: virtual keyboard 10: case 11: acquisition unit 12: storage unit 13: evaluation unit 14: generation unit 15: output unit 16: update unit 100: estimation System 101: CPU
102: ROM
103: RAM
104: Storage unit 105: I / F
106: I / F
107: I / F
108: input part 109: output part 110: internal bus S110: acquisition means S120: evaluation means S130: generation means S140: update means

Claims (9)

An estimation system for estimating a character string input using a virtual keyboard, comprising:
Acquisition means for acquiring target information including moving image data obtained by shooting an operation of pressing the virtual keyboard by the user and position information of the user's hand in the moving image data;
The past target information acquired in advance, reference information including a character string input by the past target information, and three or more stages of association between the past target information and the reference information are stored. A reference database,
An evaluation unit which refers to the reference database and acquires evaluation information including the first association degree of three or more stages between the target information and the reference information;
Generation means for generating an estimation result based on the evaluation information;
An estimation system comprising: The video data is
RGB moving image data generated using an RGB camera corresponding to a visible light region,
Infrared moving image data generated using an infrared camera corresponding to an infrared light region;
The estimation system according to claim 1, comprising: The position information is generated by using a distance measurement camera that measures a distance from an object, and includes distance data indicating a distance between the distance measurement camera and the hand of the user. An estimation system according to Item 2. The estimation system according to claim 3, wherein the projection unit that projects the virtual keyboard, the RGB camera, the infrared camera, and the distance measurement camera are mounted on the same terminal. The estimation system according to any one of claims 1 to 4, wherein the target information and the past target information include keyboard information indicating a type of the virtual keyboard. The estimation system according to any one of claims 1 to 5, wherein the target information and the past target information include linguistic information. The estimation system according to any one of claims 1 to 6, wherein the reference database is constructed by machine learning. The apparatus according to any one of claims 1 to 7, further comprising: an updating unit that reflects the relationship on the degree of association when newly acquiring a relationship between the past target information and the reference information. The estimation system described in Item 1 An estimation apparatus for estimating a character string input using a virtual keyboard, comprising:
An acquisition unit configured to acquire target information including moving image data obtained by capturing an operation of the user pressing the virtual keyboard and position information of the user's hand in the moving image data;
The past target information acquired in advance, reference information including a character string input by the past target information, and three or more stages of association between the past target information and the reference information are stored. A reference database,
An evaluation unit which refers to the reference database and acquires evaluation information including three or more stages of first association between the target information and the reference information;
A generation unit that generates an estimation result based on the evaluation information;
An estimation apparatus comprising:

Images