WO2024057375A1 - Operation panel device comprising position estimation unit based on machine learning - Google Patents

Abstract

This operation panel device comprises: a display panel (10) that has a display screen; a plurality of transparent antennas (21) to (2N) that are two-dimensionally arranged on the display screen of the display panel; a reception information generation unit 330 that generates reception information based on reception signals from the plurality of transparent antennas (21) to (2N), which are exposed to reflected waves from an object present on the display screen of the display panel (10), the reception information being associated with the transparent antennas (21) to (2N) that have received the reflected waves and corresponding to the plurality of transparent antennas (21) to (2N); a storage unit 422 that stores training information corresponding to the plurality of transparent antennas (21) to (2N) associated with the position information on the display screen of the display panel (10); and an object position estimation unit 40 that performs machine learning on the reception information generated by the reception information generation unit 330 by using the training information stored in the storage unit 422, and estimates the position of the object on the display screen of the display panel (10).

Description

[Name of the invention determined by ISA based on Rule 37.2] Operation panel device equipped with a position estimation unit using machine learning

The present disclosure relates to an operation panel device having a function of estimating the position of an object on a display screen of a display panel.

In recent years, many types of electronic devices have adopted touch-type operation panels as user interfaces (hereinafter referred to as UI).
Further, Patent Document 1 proposes a sensing screen device that is a non-contact operation panel as a UI.
The sensing screen device shown in Patent Document 1 includes a first antenna unit that transmits a sensing signal and a plurality of antenna units that receive reflected signals of the sensing signal on a transparent antenna layer disposed directly above the display screen of the sensing screen. The second antenna unit includes a second antenna unit which, when the reflected signal of the sensing signal is received, calculates the coordinates of the contact object on the surface where the sensing screen is located, based on the power of the received reflected signal and the second antenna unit that receives the reflected signal on the surface. Determine according to the coordinates of the antenna unit.

Special table 2017-527146 publication

The sensing screen device disclosed in Patent Document 1 estimates the position of a contact object using the second antenna unit that has received a high-power reflected signal among a plurality of second antenna units that have received reflected signals. Therefore, there is a problem that the accuracy of estimating the position of a contact object is poor.

The present disclosure is intended to solve the above-mentioned problems, and aims to provide an operation panel device that can estimate the position of an object on a display screen of a display panel with high accuracy.

An operation panel device according to the present disclosure includes a display panel having a display screen, a plurality of transparent antennas arranged two-dimensionally on the display screen of the display panel, and antennas that reflect light from an object present on the display screen of the display panel. a reception information generation unit that generates reception information corresponding to the plurality of transparent antennas that are linked to the transparent antenna that received the reflected wave based on reception signals from the plurality of transparent antennas that received the reflected wave; and a display. A storage unit that stores teacher information corresponding to a plurality of transparent antennas linked to position information on the display screen of the panel, and a teacher information stored in the storage unit for receiving information generated by the reception information generation unit. The apparatus includes an object position estimation unit that performs machine learning using information to estimate the position of the object on the display screen of the display panel.

According to the present disclosure, the target object position estimating unit performs machine learning using the teacher information stored in the storage unit on the received information corresponding to the plurality of transparent antennas generated by the received information generating unit, and Since the position of the object on the display screen of the display panel is estimated, the position of the object on the display screen of the display panel can be estimated with high accuracy.

1 is a block diagram showing an operation panel device according to Embodiment 1. FIG. FIG. 2 is a schematic diagram showing the surface of a panel in which a transparent antenna array is embedded in the operation panel device according to the first embodiment. 3 is an explanatory diagram showing up-chirp signals Tx(1) to Tx(N) generated by a signal generator in the operation panel device according to the first embodiment. FIG. 2 is a schematic front view showing the relationship between an instruction image displayed on a display screen of a panel and a finger in the operation panel device according to the first embodiment; FIG. 3 is a hardware configuration diagram that is realized by dedicated hardware and shows a first example of a target object position estimating section 40 in the operation panel device according to the first embodiment. FIG. FIG. 3 is a hardware configuration diagram of a computer realized by software, firmware, or the like, showing a second example of the object position estimating section 40 in the operation panel device according to the first embodiment. 7 is a flowchart illustrating a processing procedure of a position estimation operation in which a finger position is estimated in the operation panel device according to the first embodiment. 7 is a flowchart showing a processing procedure of a calibration execution operation for generating teacher information in the operation panel device according to the first embodiment. FIG. 7 is a schematic front view showing the relationship between an instruction image displayed on the display screen of the panel and a finger in the operation panel device according to the second embodiment. FIG. 7 is a block diagram showing an operation panel device according to a third embodiment. FIG. 7 is a block diagram showing an operation panel device according to a fourth embodiment. FIG. 7 is a block diagram showing an operation panel device according to a fifth embodiment.

Embodiment 1.
The operation panel device according to the first embodiment will be explained using FIGS. 1 to 8.
The operation panel device according to the first embodiment is a non-contact type operation panel having a calibration function, and includes a display panel (hereinafter abbreviated as panel) 10, a transparent antenna array 20, a transmitting/receiving circuit section 30, and object position estimation. It includes a section 40 and a control section 50.
That is, the operation panel device according to the first embodiment allows an object such as a finger or a stylus (hereinafter, a finger will be described as a representative object) to approach the panel 10 without physically contacting the panel 10. This is a user interface (UI) that allows you to operate the screen.

The operation panel device according to the first embodiment performs machine learning using teacher information on received information based on received signals from a plurality of transparent antennas forming a transparent antenna array 20 that have received reflected waves from a finger. It has a function of estimating the position of the finger on the display screen of No. 10, that is, a position estimation function.
Further, the operation panel device has a function of acquiring a plurality of pieces of teacher information each associated with received information based on received signals from a plurality of transparent antennas, that is, a function of performing calibration.

Panel 10 has a display screen.
The transparent antenna array 20 includes a plurality of transparent antennas 2 ₁₁ - 2 , which are two-dimensionally spread and embedded throughout the interior of the panel 10 in order to improve the estimation accuracy of the finger position by MIMO (Multiple-Input Multiple-Output). Consisting of 2 _mn .

The plurality of transparent antennas 2 ₁₁ to 2 _mn are two-dimensionally arranged in m rows and n columns over the entire display screen of the panel 10, as shown in FIG. m and n are natural numbers, and at least one of them is a natural number of 2 or more.
For convenience of explanation, the transparent antenna 2 ₁₁ will be referred to as the first transparent antenna 2 ₁ , the transparent antenna 2 ₁₂ will be referred to as the second transparent antenna 2 ₂ , and the transparent antenna 2 _mn will be referred to as the Nth transparent antenna 2 _N. That is, the transparent antenna array 20 is constituted by N transparent antennas from the first transparent antenna ₂₁ to the _Nth transparent antenna 2N. N is a natural number of 2 or more.
Furthermore, if it is not necessary to separately explain the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N , they will be explained as transparent antennas 2 _k (k=1 to N).

The transparent antenna _2k is a transmitting/receiving antenna that serves as both a transmitting antenna and a receiving antenna.
Note that the transparent antenna _2k may have separate transmitting antennas and receiving antennas. Further, while the number of receiving antennas is N, the number of transmitting antennas may be smaller than the number of receiving antennas, for example, N/2.
When the transparent antenna _2k functions as a transmitting antenna, it emits a transmitting wave consisting of a radio wave onto the display screen of the panel 10, that is, into the space where the finger is present.
The transmitted wave is, for example, a high frequency wave generated by a transmitted signal in a high frequency band such as L band, S band, C band, or X band.

When the transparent antenna _2k functions as a receiving antenna, it receives the reflected wave reflected by the finger present on the display screen of the panel 10, and outputs a received signal based on the reflected wave.
The transparent antenna _2k is an antenna created using a transparent conductor such as a metal mesh with fine wiring, an indium tin oxide film, or graphene, and is an antenna whose shape is not visible to the naked eye.

The transmitting/receiving circuit section 30 includes an input/output switching section 31, a signal transmitting section 32, and a receiving information generating section 33.
The transmitting/receiving circuit section 30 sequentially selects from the first transparent antenna ₂₁ to the Nth transparent antenna _2N every cycle, outputs a high frequency transmission signal to the selected transparent antenna _2k , and outputs a high frequency transmission signal to the selected transparent antenna 2k. The first transparent antenna _{21 to the Nth transparent antenna 2N receives} the reflected wave reflected from the finger present on the display screen of the panel 10 for each _k transmitted wave. Based on the received signal, the reception information corresponding to the transparent antenna _2k selected as the transmitting antenna and the first transparent antenna ₂₁ to Nth transparent antenna _2N associated with the transparent antenna that received the reflected wave is transmitted. Generate and output.

The input/output switching section 31 transmits the high frequency transmission signal from the signal transmitting section 32 to the transparent antenna _2k selected by the signal transmitting section 32, and transmits the high frequency transmission signal from the signal transmitting section 32 to the first transparent antenna 2k that receives the reflected wave reflected from the finger. The received signals from the _first to Nth transparent antennas _2N are transmitted to the received information generation section 33.
The input/output switching unit 31 has first switching units _{31 1} to _{Nth switching units 31 N corresponding} to the first transparent antennas 2 ₁ to Nth transparent antennas 2 N.
If it is not necessary to separately explain the first switching unit 31 ₁ to the Nth switching unit 31 _N , they will be described as switching units 31 _k (k=1 to N).

The switching unit _31k is a circulator having three ports that is generally known in the field of high frequency signals.
The switching unit _31k has an input port, an input/output port, and an output port.
The input port of the switching unit 31 _k is connected to the output end of the signal transmitting unit 32 , the input/output port is connected to the corresponding transparent antenna 2 _k , and the output port is connected to the reception information generating unit 33 .
In the switching unit _31k , the input port and the input/output port are connected when transmitting, and the input/output port and the output port are connected when receiving.

The switching unit _31k connects the transparent antenna _2k and the received information generating unit 33.
The switching unit _31k selected by the signal transmitting unit 32 connects the corresponding transparent antenna _2k and the signal transmitting unit 32, outputs the transmission signal from the signal transmitting unit 32 to the transparent antenna 2k, and then switches the transparent antenna _2k to the transparent antenna 2k. _k and the reception information generation section 33 are brought into a connected state.

The signal transmitting section 32 sequentially selects the first switching section ₃₁₁ to the Nth switching section _31N every cycle and outputs a high frequency transmission signal to the selected switching section _31k .
At this time, the selection order of the first switching unit 31 ₁ to the Nth switching unit 31 _N is the first switching unit 31 ₁ , the second switching unit 31 ₂ , . . . , the Nth switching unit 31 _N. In FIG. 2, the transparent antennas are selected from the left end to the right end in the first row, from the left end to the right end in the second row, and from the left end to the right end in the m-th row.
Note that the selection order from the first switching unit 31 ₁ to the Nth switching unit 31 _N is the Nth switching unit 31 _N , the (N-1)th switching unit 31 _N-1 , . . . The switching order may be the same as the switching unit ₃₁₁ , and the switching order does not matter.

The signal transmitter 32 includes a signal generator 321 and an output destination selector 322.
The signal generator 321 generates an up-chirp signal or a down-chirp signal according to the Frequency Modulated Continuous Wave (FMCW) method or the Fast-Chirp Modulation (FCM) method, and transmits one signal. N transmission signals are output every cycle.
The signal generator 321 repeatedly outputs N transmission signals every transmission cycle c (c=1, 2, . . . ).

In the first embodiment, as shown in FIG. 3, the signal generator 321 generates up-chirp signals Tx1 to TxN whose frequencies change over time every transmission cycle, and generates up-chirp signals Tx1 to TxN. ~TxN frequencies are upconverted to high frequency band signals, and the upconverted signals are output to the output destination selection section 322 as transmission signals TX1~TXN. Examples of high frequency bands include frequency bands such as L band, S band, C band, and X band.
The signal generator 321 generates up-chirp signals Tx1 to TxN based on a timing signal indicating time from the control unit 50.

The number N of transmission signals TX1 to TXN in one transmission cycle is the same as the number N of the first transparent antennas ₂₁ to Nth transparent antennas _2N , and in the first embodiment, the number N of transmission signals TX1 to TXN in one transmission cycle is The N-th transmission signals TX1 to TX1 are associated with the first transparent antennas ₂₁ to N-th transparent antennas _2N .
The order in which the up-chirp signals Tx1 to TxN are generated by the signal generator 321 is the same as the order in which the first transparent antenna ₂₁ to the Nth transparent antenna _2N are selected as transmitting antennas. The order is the second signal Tx2, . . . , the Nth signal TxN.

If it is not necessary to separately explain the N transmission signals TX1 to TXN, they will be described as transmission signals TXk (k=1 to N).
Note that the transmission signal TXk may be a signal based on a pulse signal instead of a signal based on a signal whose frequency changes over time, such as an up-chirp signal and a down-chirp signal.

The output destination selection unit 322 acquires the transmission signal TXk from the signal generator 321 in synchronization with the timing signal indicating the time from the control unit 50, and is connected to the transparent antenna _2k corresponding to the acquired transmission signal TXk. The switching unit _31k is selected and the transmission signal TXk is output to the selected switching unit _31k .
The output end of the output destination selection section 322 is connected to the input ports of the first switching section ₃₁₁ to the Nth switching section _31N .

The reception information generation unit 33 converts the reception signals from the first transparent antennas ₂₁ to Nth transparent antennas _2N that have received the reflected waves from the fingers present on the display screen of the panel 10 into the first switching mode. The transparent antenna 2k selected as the transmitting antenna and the transparent antenna _2k that received the reflected wave _are linked based on the received signal that is input through the N-th switching unit _31N from the unit ₃₁₁ . Reception information corresponding to the first transparent antenna ₂₁ to the Nth transparent antenna _2N is generated and output.

The reception information generating section 33 is a first signal receiving section corresponding to the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N and corresponding to the first switching section 31 ₁ to the Nth switching section 31 _N. 33 ₁ to 33 _N.
If it is not necessary to separately explain the first signal receiving section ₃₃₁ to the Nth signal receiving section _33N , they will be described as a signal receiving section _33k (k=1 to N).

The input end of the signal receiving section _33k is connected to the output port of the corresponding switching section _31k .
The signal receiving section _33k receives the received signal from the corresponding transparent antenna _2k via the corresponding switching section _31k .
The signal receiving section 33 _k includes a down-converting section that down-converts the frequency of a high frequency band of the received signal to a frequency of an intermediate frequency band, and a receiving signal that is an analog signal down-converted by the down-converting section and receives the received signal that is a digital signal. It has an analog/digital conversion section that performs A/D conversion to a signal.

The signal receiving unit _33k receives a time signal indicating the time from the control unit 50, and receives information on the transparent antenna _2k selected as the transmitting antenna, the timing at which the selected transparent antenna _2k radiated the transmission wave, The corresponding transparent antenna 2 _k receives the reflected wave from the finger, acquires the timing at which the received signal is received via the corresponding switching unit 31 _k , and also acquires the timing of receiving the received signal via the corresponding switching unit 31 _k . Obtain frequency and intensity.

Since the selection order of the first to _Nth transparent antennas _2N is determined, the time when the selected transparent antenna _2k radiates the transmission wave is determined by the signal receiving unit _33k . The time at which a transmission wave is radiated from the antenna 2 _k is input via the control unit 50 .
Further, the time at which the received signal is received is acquired by the signal receiving section _33k based on a signal indicating the time from the control section 50.

The signal receiving unit 33 _k uses the acquired information to link the transparent antenna 2 _k selected as the transmitting antenna and the transparent antenna 2 _k that received the reflected wave based on the received signal received via the corresponding switching unit 31 _k . The received information S (t, g, h) is generated as digital data and output to the object position estimating section 40.
The reception information S(t, g, h) includes information indicating the frequency and intensity of the reception signal acquired from the transparent antenna _2k . The received information S(t, g, h) may include a change in the shape of the spectrum of the received signal.

In the reception information S (t, g, h), t indicates the sampling time with the sampling start time at each reception antenna set to 0 (reception), and g is the reception time of the reflected wave, which is digital data, by the transparent antenna _2k . Transmitting antenna identification information (e.g. g=1 to N), which is digital data that identifies the (transmitting) transparent antenna _2k selected as the transmitting antenna, and h is digital data that identifies the transparent antenna _2k that received the reflected wave. This is receiving antenna identification information (eg, h=1 to N).

Now, assuming that the first transparent antenna ₂₁ is selected as the transmission antenna, the signal generator 321 outputs the transmission signal TX1, and the output destination selection section 322 selects the first switching section ₃₁₁ . The transmission signal TX1 is input to the first transparent antenna ₂₁ , and the first transparent antenna ₂₁ emits a transmission wave consisting of a radio wave onto the display screen of the panel 10, that is, the space where the finger is present.
When the k-th signal receiving unit 33 _k receives a received signal based on the reflected wave from the corresponding transparent antenna 2 _k that received the reflected wave reflected from the finger via the corresponding switching unit 31 _k , the received information S Generate (t, 1, k).

When the first transparent antenna 2 ₁ is selected as the transmitting antenna, the reception information S is generated in all of the first signal reception units 33 ₁ to 33 _N , so the reception information generation unit 33 generates N received information S(t, 1, h).
Furthermore, in one transmission cycle, since the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N are selected as transmitting antennas in order, the received information generation unit 33 generates (N×N) pieces of received information S( t, g, h) and output to the target object position estimation section 40.

The object position estimation unit 40 has two functions.
The first function is the reception information S(t, g, h) generated by the reception information generation unit 33, in this example, (N×N) pieces of reception information S(t, h) generated in one transmission cycle. This is a function that performs machine learning using teacher information for g and h) to estimate the position of the finger on the display screen of the panel 10. Note that, as is generally known, a learning model is created by performing machine learning using teacher information, and the position of the finger on the display screen of the panel 10 is estimated using the learning model.

The second function is to display an instruction image IP for placing a finger on the display screen of the panel 10, and to receive a reflected wave reflected from the finger at a setting location along the displayed instruction image IP. Based on the reception signals from the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N , reception information corresponding to the transparent antenna that received the reflected wave, that is, per transmission cycle generated by the reception information generation unit 33 Calibration using (N×N) pieces of received information S (t, g, h) as calibration information and associated with position information (x, y) on the display screen of the panel 10 using the instruction image IP This is a function of generating teacher information S(t, g, h|x, y) which is result data, that is, a calibration function.
x and y are the x and y coordinates on the display screen of the finger located at the set position along the instruction image IP that received the reflected wave.
Note that a vertical coordinate (z coordinate) may be added to the display screen of the panel 10 as position information.

The teacher information S (t, g, h | x, y) is the position information (x, y) on the display screen of the panel 10 based on the instruction image IP and the first All received information S _{( t} , _{g ,} This information is generated as a pair of h).

The object position estimation section 40 includes a reception information acquisition section 41 and a machine learning section 42.
The reception information acquisition unit 41 acquires reception information S(t, g, h) generated by each of the first signal reception units 33 ₁ to 33 _N.
The reception information acquisition unit 41 receives (N×N) information per transmission cycle c, which is generated by the reception information generation unit 33 composed of the first signal reception unit ₃₃₁ to the first signal reception unit _33N . The received information S(t, g, h) is acquired and provided to the machine learning unit 42.

The machine learning unit 42 has the above-described first function and second function.
The machine learning section 42 includes a position estimation section 421, a teacher information storage section 422, and a teacher information generation section 423.
The position estimation unit 421 receives the reception information S(t, g, h) acquired by the reception information acquisition unit 41, in this example, (N×N) pieces of reception information S(t, g, h) per one transmission cycle c. On the other hand, a plurality of teacher information S(t, g, h | Machine learning is performed using , the position (X, Y) of the finger on the display screen of the panel 10 is estimated, and the estimated position of the finger is output.

The machine learning performed by the position estimation unit 421 is machine learning using a machine learning method such as deep learning or Gaussian process regression. Instead of the machine learning method using deep learning and Gaussian process regression, other machine learning frameworks may be used.
When using deep learning as a machine learning method, the weight parameters that connect each layer correspond to teacher information.

For example, in a machine learning framework using deep learning, the training data S (t, g, h | x, y) acquired at all positions (x, y) is used as learning data input, and the true finger position is Learn a deep learning model whose output is (x, y). All weight parameters in the deep learning model are adjusted by back propagation or the like so that the deep learning model outputs a value close to the true finger position (x, y). All these weight parameters correspond to teacher information in the deep learning model.

When Gaussian process regression is used as a machine learning method, Gram matrix parameters correspond to training information.
For example, in Gaussian process regression, the degree of approximation of all pairs of training information S (t, g, h | x, y) acquired at all positions (x, y) is calculated using a default kernel function. A Gram matrix of M rows and M columns is generated using these kernel function outputs as matrix elements.
Here M is the number of all learning positions (x, y). The predetermined kernel function is, for example, a Gaussian kernel. By performing matrix operations using this Gram matrix on the received information S (t, g, h | X, Y) when a finger is at an unknown position (X, Y), the unknown finger position (X, The method for estimating Y) is a finger position estimation method using Gaussian process regression, and this Gram matrix corresponds to the teacher information.
When using other supervised machine learning frameworks, the parameters of the supervised machine learning model correspond to the supervised information.

The teacher information generation section 423 is controlled by the control section 50 in cooperation with the panel 10 and the transmitting/receiving circuit section 30 during a calibration operation to acquire a plurality of pieces of teacher information.
The teacher information generation unit 423 causes the instruction image IP to display the finger on the display screen of the panel 10 via the control unit 50.
The instruction image IP is an image for giving an instruction to the user to slowly trace the slowly advancing leading part with a finger.

As shown in FIG. 4, the instruction image IP, for example, moves from the upper left end of the display screen of the panel 10 to the upper right end, goes down one step, moves back and forth several times from the right end to the left end, and slowly draws a drawing line that ends from the lower right end to the lower left end. This is a drawn image showing a zigzag trajectory.
As shown in FIG. 4, the leading portion of the drawn line that is the instruction image IP moves slowly on the display screen of the panel 10, and the moved portion becomes a darker drawn line IPb. The thin drawn line IPa represents the drawn line from which the dark drawn line IPb extends.
The drawn line, which is the instruction image IP, indicates an instruction that the user slowly traces with his or her finger on the display screen of the panel 10, following the movement of the leading part.
It takes several tens of seconds for the leading portion to move all over the thin drawing line IPa.

The teacher information generation unit 423 generates a plurality of setting positions P _d (d=1 to D) at equal intervals in the drawing line that is the instruction image IP, that is, from the first setting position P ₁ to At each of the D-th setting positions _PD , (N×N) pieces of reception information S(t, g, h) per one transmission cycle c generated by the reception information generation unit 33 are acquired as calibration information. D is a natural number of 2 or more.
That is, at each setting position P _d , the teacher information generating unit 423 inputs the transmission signal from the signal transmitting unit 32 to the selected transparent antenna 2 _k via the switching unit 31 _k , and inputs the transmission signal from the signal transmitting unit 32 to the selected transparent antenna 2 _k . A transmission wave is emitted from the

The first transparent antenna ₂ receives the reflected wave reflected from the finger in response to the transmitted wave from the selected transparent antenna 2 _k.The first transparent antenna 2 receives the reflected wave based on the received signal from the 1st to Nth transparent antennas 2 _N. The teacher information generation unit 423 acquires the reception information corresponding to the transparent antenna, that is, the reception information S(t, g, h) generated by the reception information generation unit 33, via the reception information acquisition unit 41.
The teacher information generation unit 423 slowly draws a zigzag trajectory from the beginning (upper left end) to the end (lower left end) of the leading part of the drawing line that is the instruction image IP, and sets the set position P of the drawn line where the leading part is located. _d , the reception information S(t, g, h) generated by the reception information generation section 33 is acquired via the reception information acquisition section 41.

At each set position _Pd , in one transmission cycle c, the first transparent antenna ₂₁ to the Nth transparent antenna _2N are sequentially selected as transmitting antennas, and the teacher information generation unit 423 transmits the information via the reception information acquisition unit 41. (N×N) pieces of reception information S(t, g, h) are acquired from the reception information generation unit 33.
Therefore, until the finger on the display screen of the panel 10 finishes tracing the drawing line following the beginning of the drawing line that is the instruction image IP, the first setting position P ₁ to the Dth setting position P _D In all cases, the teacher information generation unit 423 acquires (N×N) received information S(t, g, h) from the reception information generation unit 33 via the reception information acquisition unit 41.

At each setting position P _d , the teacher information generation unit 423 generates a panel at the beginning of the instruction image IP at the setting position P _d in which the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N are sequentially selected as transmitting antennas. The calibration result data is a pair of position information (x _d , y _d ) on the display screen of 10 and (N×N) received information S (t, g, h) from the received information generation unit 33. Information S (t, g, h | x _d , y _d ) is generated, and teacher information S (t, g, h | x _d , y _d ) is stored in the teacher information storage unit 422 .

That is, the teacher information generation unit 423 generates the positional information (x _d , y _d ) (d=1 to D) of the leading part in the instruction image IP, that is, the position information ( x _d , y _d ) is linked to the teacher information S(t, g, h | from the setting position P ₁ to the Dth setting position P _D , respectively.
Note that the positional information of the leading portion is expressed by x-axis and y-axis coordinates on the display screen of the panel 10. Furthermore, the coordinates of the z-axis in the direction perpendicular to the display screen of the panel 10, that is, the xy plane, may be added as the position information of the leading part.

Next, a specific configuration of the target object position estimating section 40 will be explained.
As a first example, each of the reception information acquisition section 41 and the machine learning section 42 is realized by dedicated hardware including a reception information acquisition circuit and a machine learning circuit, as shown in FIG.
The reception information acquisition circuit corresponds to the reception information acquisition section 41, and the machine learning circuit corresponds to the machine learning section 42.

The reception information acquisition circuit is an interface circuit that transmits the reception information S (t, g, h) from the reception information generation section 33 to the machine learning circuit, and a generally known interface circuit can be used.
The machine learning circuit includes a position estimation circuit that constitutes a position estimation section 421 , a memory that is a teacher information storage section 422 , and a teacher information generation circuit that constitutes a teacher information generation section 423 .
Each of the position estimation circuit and the teacher information generation circuit is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or This is achieved by a combination of these.

As a second example, the reception information acquisition section 41 and the machine learning section 42 are realized by a computer incorporating a program of software, firmware, or a combination of software and firmware, as shown in FIG.
FIG. 6 shows a typical example of a computer, which includes a CPU (Central Processing Unit), an input interface section, a RAM (Random Access Memory), a ROM (Read Only Memory), and an output interface section.
The CPU controls and manages the input interface section, RAM, ROM, and output interface section.

The CPU functions as a position estimator 421 and a teacher information generator 423 in cooperation with the ROM and RAM.
The ROM includes a position estimation program that executes a position estimation process to function as the position estimation unit 421, and a position estimation program that executes a process procedure to generate teacher information to function as the teacher information generation unit 423. A teacher information generation program, which is a calibration execution program, is stored.
ROM is a type of recording medium, and is a DVD (Digital Versatile Disc), CD (Compact Disc), HDD (Hard Disk Drive), or USB memory.
The RAM functions as the teacher information storage section 422, and also plays the role of temporarily storing the program stored in the ROM when the CPU executes the program stored in the ROM.

The input interface section corresponds to the reception information acquisition section 41 that acquires the reception information S (t, g, h) from the reception information generation section 33, and when the computer performs the calibration execution operation, the panel 10 based on the instruction information. The position information (x, y) of the leading part on the display screen of is acquired.
When the computer performs a position estimation operation, the output interface unit outputs the estimated finger position, which is the estimation result by the CPU, ROM, and RAM functioning as the position estimation unit 421, and when the computer performs a calibration operation, the output interface unit The CPU, ROM, and RAM functioning as the information generation unit 423 output instruction image information and the like to display the instruction image IP on the display screen of the panel 10.

Although the computer uses a CPU, it is also possible to use a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or hardware that executes a DSP (Digital Signal Processor) program. Good too.
Further, the present invention is not limited to the first and second examples, and some of the components may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, or the like.

Next, the operation of the operation panel device according to the first embodiment will be explained.
First, the position estimation operation for one transmission cycle c (c=1) will be explained using FIG. 7. Although the transmission cycle c is performed multiple times, the position estimation operation is the same, so only one transmission cycle c (c=1) will be described.
In step ST11, the transparent antenna _2k selected as the transmitting antenna is initially set as the first transmitting antenna ₂₁ (k=1), and in step ST12, in order to radiate a transmitted wave from the first transparent antenna ₂₁ , The output destination selection section 322 selects the first switching section 311 to output the first transmission signal TX1 from the signal generator ₃₂₁ to the first transparent antenna ₂₁ .

The first transparent antenna ₂₁ radiates a transmitted wave into the space where the finger is present.
The transmission wave radiated from the first transparent antenna ₂₁ is reflected by the finger, and the first to Nth _transparent antennas _2N receive the reflected wave by the finger (step ST13).
The received signals from the reflected waves received by the first transparent antennas 2 ₁ to Nth transparent antennas _2N are transmitted to the first signal receiving unit via the first switching unit ₃₁₁ to the Nth switching unit _31N , respectively. ₃₃₁ to the first signal receiving section _33N .

Each of the first signal receiving units _{33_1} to _{33_N} performs a receiving process on each of the input receiving signals, and generates receiving information S(t, 1, h( ₁ to N)) based on the received receiving signals, which indicates that the transparent antenna selected as the transmitting antenna is the first transparent antenna _{2_1} , the time when the first transparent antenna _{2_1} emitted a transmitting wave, and the time when the corresponding transparent antenna 2_k received a reflected wave from the finger (step ST14).

Reception information S (t, 1, h (1 to N)) when the first transparent antenna 2 ₁ is selected as the transmitting antenna, respectively, from the first signal receiving unit 33 ₁ to the first signal receiving unit 33 _N Once generated, the process proceeds to step ST15, where the transparent antenna _2k selected as the transmitting antenna is set as the second transmitting antenna ₂₂ (k=2), and the process returns to step ST12.
In step ST12, in order to radiate a transmission wave from the second transmission antenna 2 ₂ (k=2) as a transmission antenna, the output destination selection unit 322 switches the second transmission signal TX2 from the signal generator 321 to a second switching mode. ₃₁₂ and outputs it to the second transparent antenna ₂₂ .

A second transparent antenna ₂₂ radiates a transmitted wave into the space where the finger is present.
The transmission wave radiated from the second transparent antenna ₂₂ is reflected by the finger, and the first to _Nth transparent antennas _2N receive the reflected wave by the finger (step ST13).
The received signals from the reflected waves received by the first transparent antennas 2 ₁ to Nth transparent antennas _2N are transmitted to the first signal receiving unit via the first switching unit ₃₁₁ to the Nth switching unit _31N , respectively. ₃₃₁ to the first signal receiving section _33N .

Each of the first signal receiving units 33 ₁ to 33 _N performs reception processing on each input received signal, and the transparent antenna selected as the transmitting antenna is used as the second transparent antenna 2 _{2 ,} the reception information _S ( t, 2, h (1 to N)) (step ST14).

In this way, the loop from step ST12 to step ST15 is repeated until the Nth transparent antenna _2N is selected as the transmitting antenna, the transparent antenna _2k is selected as the transmitting antenna in order, and the loop from step ST12 to step ST15 is repeated. (N×N) pieces of received information S(t, g, h) are generated by the receiving unit ₃₃₁ to the first signal receiving unit _33N .

In step ST14, the reception information generation unit 33 generates a first transparent antenna 2 two-dimensionally arranged on the display screen of the panel 10 that receives a reflected wave reflected from a finger present on the display screen of the panel 10. ₁ to N-th transparent antennas 2 Transparent antennas ₂ to which a received signal from _N is input and which receives a reflected wave based on the input received _signals. This is a step of generating reception information S(t, g, h) corresponding to the transparent antenna _2N .

If the Nth transparent antenna _2N is selected as the transmitting antenna in step ST15, the process proceeds to step ST16.
In step ST16, the reception information acquisition unit 41 acquires (N×N) reception information S(t, g, h) from the reception information generation unit 33, and (N×N) reception information S(t , g, h) to the position estimation unit 421.

The position estimation unit 421 calculates a plurality of position information (x, y) of the display screen of the panel 10 stored in the teacher information storage unit 422 for the (N×N) pieces of received information S (t, g, h). Machine learning is performed using a plurality of pieces of teacher information S (t, g, h | Output the estimated finger position.
Note that the subsequent estimation of the finger position is performed using a learning model generated as a result of machine learning.

In step ST16, the position estimating unit 421 in the target object position estimating unit 40 inputs the (N×N) received information S(t, g, h) generated by the received information generating unit 33 into the teacher information storage unit. This is a step of performing machine learning using the teacher information S (t, g, h | x, y) stored in 422 to estimate the position (X, Y) of the finger on the display screen of the panel 10.

Further, when the reception information acquisition section 41 and the machine learning section 42 are realized by a computer, a program for executing the process of step ST16 is stored in the ROM, and the CPU executes the program stored in the ROM.
The program stored in the ROM includes first transparent antennas 2 1 to 1 that are two-dimensionally arranged on the display screen of the panel 10 that receive reflected waves from a finger present on the display screen of the panel ₁₀ . _N 's transparent antenna 2 A first transparent antenna ₂ associated with the transparent antenna that received the reflected wave based on the received signal from _N. Receiving information S(t, g, h) and machine learning using the teacher information S (t, g, h | x, y) on the acquired received information S (t, g, h), and This is a target object position estimation program for estimating the position of a target object on the display screen of an operation panel device, which includes a procedure for estimating the position (x, y) on the display screen.

Next, the calibration execution operation in the operation panel device according to the first embodiment, that is, the teacher information S( t, g, h|x, y) will be described using FIG. 8.
The teacher information generating section 423 is controlled by the control section 50 in cooperation with the panel 10 and the transmitting/receiving circuit section 30.

In step ST21, the teacher information generation unit 423 outputs instruction image information for displaying the instruction image IP on the display screen of the panel 10 to the panel 10 via the control unit 50.
As shown in FIG. 4, the panel 10 displays a zigzag thin drawing line IPa on the display screen, and the leading part slowly moves along the thin drawing line IPa, and the moved part becomes a dark drawing line IPb. As a result, an instruction image IP indicating an instruction for the user to slowly trace with a finger following the movement of the beginning of the dark drawn line IPb is displayed (step ST22).

When the beginning of the dark drawn line IPb in the instruction image IP reaches the first setting position _P1 , the position information (x ₁ , y ₁ ) of the beginning of the instruction image IP at the first setting position P ₁ is used as teacher information. The generation unit 423 acquires it (step ST23).
At the same time, the teacher information generation unit 423 generates the first transparent _{antenna 21 to} the Nth transparent antenna _{21 to N} 's transparent antenna 2 Acquires (N×N) pieces of reception information S(t, g, h) generated by the reception information generation section 33 based on the reception signal received by the N through the reception information acquisition section 41. (step ST24).

In step ST24, the (N×N) pieces of reception information S(t, g, h) acquired by the teacher information generation unit 423 are generated by the reception information generation unit 33 in steps ST11 to ST15 in the position estimation operation ( This is the same as N×N) pieces of received information S(t, g, h).
That is, although a detailed explanation will be omitted, at the first set position _P1 , all of the first transparent antenna ₂₁ to the Nth transparent antenna _2N are selected as transmitting antennas in order, and steps ST12 to ST15 are performed. The loop is repeated, and (N×N) received information S(t, g, h) generated by the first signal receiving unit ₃₃₁ to the first signal receiving unit _33N is used to generate teacher information. The unit 423 obtains the information.

In step ST25, the teacher information generation unit 423 generates the acquired position information (x ₁ , y ₁ ) of the first setting position P ₁ and the (N×N) received information at the acquired first setting position P _1. Teacher information S(t, g, h|x ₁ , y ₁ ) at the first setting position P ₁ is generated using S(t, g, h) as a pair.
The teacher information generation unit 423 stores the generated teacher information S (t, g, h | x ₁ , y ₁ ) at the first setting position P ₁ in the teacher information storage unit 422 (step ST26), and proceeds to step ST27. .

Next, when the head of the dark drawn line IPb in the instruction image IP reaches the second setting position _P2 , the process returns to step _ST23 , and the position information (x ₂ , y ₂ ) is acquired by the teacher information generation unit 423.
Thereafter, in step ST24, the teacher information generation unit 423 sets the first transparent antenna at the second setting position _P2 to _{the first} transparent antenna when each of the first to Nth transparent antennas _2N is used as a transmitting antenna. (N×N) pieces of reception information S(t, g, h) generated by the reception information generation unit 33 are obtained from the reception signals received by the _Nth transparent antennas 2N from ₂₁ .

In step ST25, the teacher information generation unit 423 generates the acquired position information (x ₂ , y ₂ ) of the second setting position P ₂ and the (N×N) received information at the acquired second setting position P ₂ . _The teacher information generation unit ₄₂₃ generates teacher _information S (t, g, h | _The teacher information S (t, g, _{h |}

In this way, from step ST23 to step ST23 until the beginning of the dark drawn line IPb in the instruction image IP reaches the D-th setting position P- _D , from the first setting position _P1 to the D-th setting position P- _D . The loop of step ST27 is repeated, and the teacher information S(t, g, h|x _d , y _d ) at each setting position P _d is stored in the teacher information storage section 422.
In step ST27, d of the setting position P _d is D, that is, the position set over the entire area of the instruction image IP on the display screen of the panel 10, that is, the first setting position P ₁ to the Dth setting. When the teacher information S(t, g, h|x, y) is generated for all positions _PD and stored in the teacher information storage section 422, the operation of generating the teacher information is completed.

In steps ST21 to ST27, the teacher information generation unit 423 provides the panel 10 with instruction image information for displaying an instruction image IP that causes a finger to be present on the display screen of the panel 10, and The first transparent antenna 2 receives the reflected waves reflected from the fingers existing along the image IP. The _first to Nth transparent antennas 2 receive the received signals from _N , and receive the reflected waves based on the received received signals. Using the received information S (t, g, h) corresponding to the transparent antenna that This is a step of generating the teacher information S(t, g, h|x, y) to be stored.

Further, when the reception information acquisition section 41 and the machine learning section 42 are realized by a computer, a program for executing the processes from step ST21 to step ST27 is stored in the ROM, and the CPU executes the program stored in the ROM.
The program stored in the ROM provides the panel 10 with instruction image information for displaying an instruction image IP that causes the finger to be present on the display screen of the panel 10, and causes the finger to exist along the instruction image IP displayed on the display screen of the panel 10. The reception information _S ( _t , g , h) to generate teacher information S(t, g, h | x, y) associated with position information (x, y) on the display screen of the panel 10 based on the instruction image IP. , is a teacher information generation program for estimating the position of an object on the display screen of an operation panel device.

As described above, the operation panel device according to the first embodiment includes the first transparent antenna 21 to the Nth transparent antenna _2N arranged two _- dimensionally on the display screen of the panel 10, The received information generation unit 33 generates reflected waves based on the received signals from the first transparent antennas ₂₁ to Nth transparent antennas _2N that have received the reflected waves from the fingers present on the display screen of the panel 10. The first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N associated with the transparent antenna that received Machine learning using teacher information S (t, g, h | x, y) stored in teacher information storage unit 422 is performed on received information S (t, g, h) generated by information generation unit 33. Since the position of the finger on the display screen of the panel 10 is estimated by performing the above steps, the accuracy of estimating the position of the finger is improved without the finger touching the display screen of the panel 10.

In the operation panel device according to the first embodiment, the teacher information generation unit 423 provides the panel 10 with instruction image information for displaying an instruction image IP that causes a finger to be present on the display screen of the panel 10, and displays the instruction image information on the display screen of the panel 10. A first transparent antenna 2 that has received a reflected wave reflected from a finger existing along the displayed instruction image IP.A transparent antenna that has received a reflected wave based on a reception signal from the _1st to _Nth transparent antennas 2N. Teacher information stored in the teacher information storage unit 422, in which position information (x, y) on the display screen of the panel 10 based on the instruction image IP is linked to received information S (t, g, h) corresponding to S (t, g, h | x, y) is generated, so it does not take a long time to generate and obtain the teacher information S (t, g, h | x, y), and it can be easily and easily realized. .
Furthermore, even if cracks or dirt adhere to the display screen of the panel 10, the teacher information S (t, g, h | It can be prevented.

Embodiment 2.
An operation panel device according to Embodiment 2 will be described using FIG. 9.
The teacher information generating unit 423 in the operation panel device according to the first embodiment generates reflected waves based on the received signals from all of the first transparent antennas ₂₁ to Nth transparent antennas _2N that have received the reflected waves reflected from the fingers. Teacher information S(t, g, h|x, y) is generated based on received information S(t, g, h) corresponding to the transparent antenna that received the wave.

On the other hand, in the operation panel device according to the second embodiment, the teacher information generation unit 423 generates instructions displayed on the display screen of the panel 10 among the first transparent antennas ₂₁ to _Nth transparent antennas 2N. Reception information S (t, g, h ) is used to generate the teacher information S(t, g, h|x, y).
The operation panel device according to the second embodiment is different from the operation panel device according to the first embodiment in this point, and is the same in other points.
Note that in FIG. 9, the same reference numerals as those given in FIGS. 1 to 8 indicate the same or equivalent parts.

Below, differences from the operation panel device according to Embodiment 1 will be mainly explained.
In the operation panel device according to the second embodiment, the position estimation operation is the same as the position estimation operation in the operation panel device according to the first embodiment, so a description thereof will be omitted.
Therefore, the calibration execution operation, that is, the teacher information S (t, g, h | x, y) which is the calibration result data linked with the position information (x, y) on the display screen of the panel The generation operation will be explained with reference to FIG.

In the operation panel device according to the second embodiment, similar to steps ST21 to ST23 in the operation panel device according to the first embodiment, instruction image information for displaying the instruction image IP on the display screen of the panel 10 is generated as teacher information. The unit 423 outputs the output to the panel 10 via the control unit 50 (step ST21), and as shown in FIG. The leading part moves slowly and the moved part becomes a dark drawn line IPb, thereby creating an instruction image IP that indicates an instruction for the user to slowly trace with a finger following the movement of the leading part of the dark drawn line IPb. Display (step ST22).

When the head of the dark drawn line IPb in the instruction image IP reaches the first setting position _P1 , the position information (x ₁ , y ₁ ) of the head of the instruction image IP at the first setting position P ₁ is The teacher information generation unit 423 acquires it (step ST23).
According to instructions from the teacher information generation unit 423, a plurality of transparent antennas existing around the first set position _P1 among the first transparent antenna ₂₁ to the Nth transparent antenna _2N , in this example, for example, The four surrounding transparent antennas are activated, ie, turned on. For example, in FIG. 9, as shown by the solid line frame, at the setting position P _d , there are four transparent antennas surrounding the setting position P _d : transparent antenna 2 ₁₂ , transparent antenna 2 ₁₃ , transparent antenna 2 ₂₂ , and transparent antenna 2 ₂₃ . is activated.

Among the first transparent antennas ₂₁ to Nth transparent antennas _2N , the transparent antennas other than the four activated transparent antennas are inactivated, that is, turned off. In FIG. 9, it is indicated by a broken line frame.
Note that in this example, the activation and deactivation of the transparent antenna is not limited to turning the transparent antenna on and off, but rather, the activation and deactivation of the transparent antenna is not limited to turning the transparent antenna on and off. This includes turning the receiver on and off.

At the first setting position _P1 , corresponding to step ST24 in the operation panel device according to the first embodiment, a plurality of activated transparent antennas from the first transparent antenna ₂₁ to the Nth transparent antenna _2N are For example, (4×4) pieces of received information S(t, g, h) is acquired by the teacher information generation unit 423 via the reception information acquisition unit 41.

Hereinafter, similarly to step ST25 and step ST26 in the operation panel device according to the first embodiment, the teacher information generation unit 423 uses the acquired position information (x 1 , y 1 ) of the first setting position P ₁ and the acquired position information (x ₁ , y ₁ ). Teacher information S(t _, g, h|x _{1 ,} y ₁ ), _the generated teacher information S (t, _g , h _| move on.

Next, when the beginning of the dark drawn line IPb in the instruction image IP reaches the second setting position P2, the position information (x ₂ , y _{2 ) of the beginning of the instruction image IP at the second setting position P 2 is calculated} . The teacher information generation unit 423 acquires _and generates the teacher information S(t, _g , _{h |} t, g, h|x ₂ , y ₂ ) is generated and stored in the teacher information storage unit 422.
That is, a plurality of transparent antennas existing around the second set position _P2 are activated, and received information S(t, g, h) based on the received signal from the activated transparent antenna and the second set position Teacher information S(t, g, h|x ₂ , y ₂ ) at the second setting position P ₂ is generated using the position information (x ₂ , y ₂ ) of P ₂ as a pair.

In this way, each setting position is moved in the order of setting positions from the first setting position P1 to the Dth setting position _PD until the beginning of the dark drawn line _IPb in the instruction image IP reaches _{the Dth} setting position PD. Teacher information S ₍ t _, g, _{h |}
Teacher _information S(t, _g , h | x , y) is generated and stored in the teacher information storage unit 422, the operation of generating the teacher information is completed.

In the operation panel device of embodiment 2, the step of generating teacher information S(t, g, h|x, y) is a step in which the teacher information generating unit 423 provides the panel 10 with instruction image information for displaying an instruction image IP for indicating a finger to be present on the display screen of the panel 10, and generates teacher information S(t, g, h|x, _y ) to be stored in the teacher information storage unit 422, in which position information (x, _y) on the display screen of the panel 10 due to the instruction image IP is linked to received information S(t, g, h) corresponding to a transparent antenna among the first transparent antenna 2-1 to the Nth transparent antenna 2-N that is present around the position on the display screen of the panel 10 indicated by the instruction image IP displayed on the display screen of the panel 10 and that has received a reflected wave reflected from a finger present along the instruction image IP displayed on the display screen of the panel 10, receives received signals from multiple activated antennas, and receives a reflected wave based on the received received signal.

Further, when the received information acquisition unit 41 and the machine learning unit 42 are realized by a computer, a process of generating teacher information S (t, g, h | x, y) to be stored in the teacher information storage unit 422 is executed. A program is stored in the ROM, and the CPU executes the program stored in the ROM.
The program stored in the ROM provides the panel 10 with instruction image information for displaying an instruction image IP that causes the finger to be present on the display screen of the panel 10;
Display of the panel 10 among the first transparent antennas 2 ₁ to Nth transparent antennas 2 _N that have received reflected waves from the fingers existing along the instruction image IP displayed on the display screen of the panel 10 Receive received signals from a plurality of activated antennas existing around the position on the display screen of the panel 10 pointed to by the instruction image IP displayed on the screen, and receive reflected waves used in the received received signals. The received information S (t, g, h) corresponding to the transmitted transparent antenna is stored in the teacher information storage unit 422 in which position information (x, y) on the display screen of the panel 10 based on the instruction image IP is linked. This is a teacher information generation program for estimating the position of an object on the display screen of an operation panel device, which includes a procedure for generating teacher information S (t, g, h | x, y).

As described above, the operation panel device according to the second embodiment has the same effects as the operation panel device according to the first embodiment.
Further, in the operation panel device according to the second embodiment, the teacher information generation unit 423 generates an instruction image IP displayed on the display screen of the panel 10 among the first transparent antennas 21 to _Nth transparent antennas _2N . A plurality of activated antennas exist around the position on the display screen of the panel 10 pointed by the finger and correspond to the transparent antenna that received the reflected wave based on the received signal that received the reflected wave reflected from the finger. Teacher information S(t, g, h) stored in the teacher information storage unit 422 is associated with position information (x, y) on the display screen of the panel 10 based on the instruction image IP. , g, h | The information S(t, g, h | x, y) can be generated efficiently, and the memory capacity in the teacher information storage section 422 can be used efficiently.

Embodiment 3.
An operation panel device according to Embodiment 3 will be described using FIG. 10.
The operation panel device according to the third embodiment is different from the operation panel device according to the first embodiment in that a reception sensitivity determining section 43 is further provided, and other points are the same.
The reception sensitivity determination unit 43 detects the reception sensitivity of the wave reflected from the finger by the transparent antenna _2k , and determines the teacher information S (t, g, h | x, y) when the reception sensitivity becomes less than the set value. Encourage reacquisition.
By detecting the deterioration of the reception sensitivity, the transparent antenna _2k detects the reflection from the finger due to the presence of dirt on the display screen of the panel 10 or the occurrence of cracks on the display screen of the panel 10. You can know the deterioration of wave receiving sensitivity.
Note that in FIG. 10, the same reference numerals as those given in FIGS. 1 to 8 indicate the same or equivalent parts.

Below, differences from the operation panel device according to Embodiment 1 will be mainly explained.
In the operation panel device according to the third embodiment, the position estimation operation and the calibration execution operation, that is, the teacher whose position information (x, y) on the display screen of the panel 10 is associated with calibration result data. The operation of generating the information S(t, g, h | x, y) is the position estimation operation according to the flowchart shown in FIG. 7 and the calibration according to the flowchart shown in FIG. Since this is the same as the execution operation, the explanation will be omitted.

Therefore, the receiving sensitivity determination unit 43 detects the deterioration of the receiving sensitivity of the reflected wave from the finger by the transparent antenna _2k , and when the receiving sensitivity becomes below the set value, the teacher information S(t, g, h|x, We will mainly explain the point of encouraging reacquisition of item y).
The reception sensitivity determination unit 43 uses the reception information S(t, g, h) acquired by the reception information acquisition unit 41, in this example, (N×N) pieces of reception information S(t, g, h) per one transmission cycle c. ) to obtain.

The reception sensitivity determination unit 43 detects the strength of the reception signal acquired from the transparent antenna 2 _k included in the reception information S (t, g, h) that has been acquired, and determines the strength of the reception signal based on the detected strength of the reception signal. The teacher information generation unit 423 and the control unit 50 generate reception sensitivity deterioration information that displays an image that prompts the reacquisition of the teacher information S (t, g, h | x, y) when the value becomes less than a set value. The information is applied to the panel 10 to display an image on the display screen of the panel 10 that prompts the user to reacquire the teacher information.
Note that the reception sensitivity determination section 43 may provide the reception sensitivity deterioration information to the panel 10 via the control section 50 without going through the teacher information generation section 423.

The comparison between the strength of the received signal and the set value in the reception sensitivity determination section 43 is performed, for example, as follows.
The maximum value of the strength of the received signal included in each of the (N×N) received information S (t, g, h) per one transmission cycle c is detected, and the maximum value of the strength of the detected received signal is set. Compare values.

Transmission waves are radiated from the transparent antenna _2k located near the position of the finger on the display screen of the panel 10, and the transparent antenna _2k located near the position of the finger on the display screen of the panel 10 is emitted from the finger. When receiving a reflected _wave of , h) is the maximum value of the strength of the received signal.

Therefore, if the maximum value of the strength of the received signal obtained based on the (N×N) pieces of received information S(t, g, h) becomes less than the set value, it means that the display screen of the panel 10 where the finger is present. It can be presumed that dirt has adhered or cracks have occurred on the display screen of the panel 10, and as a result, it can be determined that the receiving sensitivity of the transparent antenna _2k located near the position of the finger on the display screen of the panel 10 has deteriorated.

Note that the transparent antenna 2 _k as a transmitting antenna located near the position where the finger is present and the transparent antenna 2 _k as the receiving antenna located near the position where the finger is present are not necessarily the same.
In addition, there is not necessarily one received information S(t, g, h) that has the maximum value among the received signal strengths obtained from (N×N) pieces of received information S(t, g, h).

In the above example, the receiving sensitivity is determined by the strength of the received signal, but the spectral shape of the received signal is detected and the change in the spectral shape exceeds the set value. In other words, the spectral shape is less than or equal to the set spectral shape. In this case, reception sensitivity deterioration information may be generated that prompts reacquisition of the teacher information S(t, g, h|x, y).
The reception sensitivity determination section 43 may be incorporated into the computer together with the reception information acquisition section 41 and the machine learning section 42.
The reception sensitivity determination unit 43 also includes a sensor that detects the strength of the reception signal from the transparent antenna _2k , and compares the strength of the reception signal obtained by the sensor with a set value to obtain reception sensitivity deterioration information. But that's fine.

When an image prompting re-acquisition of teacher information is displayed on the display screen of the panel 10, the user causes the operation panel device to acquire the teacher information.
The calibration execution operation in the operation panel device, that is, the teacher information S(t, g, h | x, y ) is the same as the operation described in Embodiment 1, and is executed according to the flowchart shown in FIG.

However, the teacher information S (t, g _{, h} | When stored in _the teacher information storage section 422, the _teacher information S(t, g, h|x , y).

In this way, when the reception sensitivity of the reflected wave from the finger by the transparent antenna 2 _k deteriorates, the teacher information S (t, g, h | x, y) stored in the teacher information storage section 422 can be rewritten. Therefore, the position of the finger can be estimated using the optimal teacher information S(t, g, h|x, y) according to the state on the display screen of the panel 10.

As described above, the operation panel device according to the third embodiment has the same effects as the operation panel device according to the first embodiment.
Further, in the operation panel device according to the third embodiment, when the reception sensitivity determination unit 43 determines that the reception sensitivity of the reception signals from the first transparent antenna ₂₁ to the Nth transparent antenna _2N is equal to or less than the set value, Since reception sensitivity deterioration information is given to the panel 10 to display an image prompting the acquisition of teacher information S (t, g, h | x, y) on the display screen of the panel 10, the teacher information is The teacher information S(t, g, h | |x, y), the position of the finger can be estimated.

Embodiment 4.
An operation panel device according to Embodiment 4 will be explained using FIG. 11.
The operation panel device according to the fourth embodiment is different from the operation panel device according to the first embodiment in that an elapsed time determination section 44 is further provided, and other points are the same.
Note that in FIG. 11, the same reference numerals as those shown in FIGS. 1 to 8 indicate the same or equivalent parts.

Due to the influence of the usage environment of the operation panel device, after a long _period of time, the received information _S (t, g , h) may be affected, and the accuracy of machine learning performed by the position estimation unit 421 using the teacher information S(t, g, h | x, y) stored in the teacher information storage unit 422 may decrease. be.
Based on this point, the elapsed time determination unit 44 causes the teacher information generation unit 423 to generate all the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422, and When the elapsed time from the time stored in 422 exceeds the set time, the display screen of the panel 10 prompts to obtain the teacher information S(t, g, h|x, y).

Below, differences from the operation panel device according to Embodiment 1 will be mainly explained.
In the operation panel device according to the fourth embodiment, the position estimation operation and the calibration execution operation, that is, the teacher whose position information (x, y) on the display screen of the panel 10 is associated with calibration result data. The operation of generating the information S(t, g, h | x, y) is the position estimation operation according to the flowchart shown in FIG. 7 and the calibration according to the flowchart shown in FIG. Since this is the same as the execution operation, the explanation will be omitted.

Therefore, the elapsed time determination unit 44 determines whether the teacher information S The point of encouraging reacquisition of (t, g, h | x, y) will be mainly explained.
The elapsed time determining unit 44 receives from the teacher information generating unit 423 the time when the teacher information generating unit 423 completes the operation of generating the teacher information S (t, g, h | x, y) (hereinafter referred to as calibration completion time). and store the calibration completion time.

The elapsed time determination unit 44 periodically reads the current time, compares the read current time with the stored calibration completion time, and determines that the elapsed time from the calibration completion time to the current time is equal to or greater than the set time. Elapsed time image information for displaying an image that prompts reacquisition of teacher information S (t, g, h | x, y) is given to the panel 10 via the teacher information generation unit 423 and the control unit 50, An image indicating the need to reacquire teacher information is displayed on the screen.
Note that the elapsed time determination section 44 may provide the elapsed time image information to the panel 10 via the control section 50 without going through the teacher information generation section 423.

In the above example, the elapsed time is obtained using the calibration completion time and the current time, but the elapsed time determination unit 44 includes a timer, the timer is set according to the calibration completion time, and the elapsed time is set by the timer. When the time comes, elapsed time image information may be output.
The elapsed time determination unit 44 may be incorporated into the computer together with the reception information acquisition unit 41 and the machine learning unit 42.

When an image prompting re-acquisition of teacher information is displayed on the display screen of the panel 10, the user causes the operation panel device to acquire the teacher information.
The calibration execution operation in the operation panel device, that is, the teacher information S(t, g, h | x, y ) is the same as the operation described in Embodiment 1, and is executed according to the flowchart shown in FIG.

However, the teacher information S (t, g _{, h} | When stored in _the teacher information storage section 422, the _teacher information S(t, g, h|x , y).

Further, when the teacher information generation unit 423 completes the operation of re-acquiring the teacher information S (t, g, h | x, y), the elapsed time determination unit 44 receives the calibration completion time from the teacher information generation unit 423, Rewrite the calibration completion time.
If the elapsed time determination unit 44 has a timer, it resets the time of the timer.

As described above, the operation panel device according to the fourth embodiment has the same effects as the operation panel device according to the first embodiment.
Further, in the operation panel device according to the fourth embodiment, the teacher information generation unit 423 generates teacher information S (t, g, h | x, y) to be stored in the teacher information storage unit 422, and When the elapsed time from the time stored in 422 exceeds the set time, the panel displays elapsed time image information to display an image prompting the acquisition of teacher information S (t, g, h | x, y) on the display screen of the panel 10. 10, the teacher information S (t, g, h | Taking this into consideration, the finger position can be estimated using the optimal teacher information S(t, g, h|x, y).

Embodiment 5.
An operation panel device according to Embodiment 5 will be described using FIG. 12.
The operation panel device according to the fifth embodiment is different from the operation panel device according to the first embodiment in that a device temperature determination section 45 and a device temperature measurement section 46 are further provided, and other points are the same. It is.
Note that in FIG. 12, the same reference numerals as those given in FIGS. 1 to 8 indicate the same or equivalent parts.

In the operation panel device, when the temperature change of the device exceeds the set value, the device temperature drift from the first transparent antenna ₂₁ to the Nth transparent antenna _2N exceeds the specified value, and the The received information S (t, g, h) by the received signals from the 1st transparent antenna 2 ₁ to the Nth transparent antenna 2 _N is influenced, and the teacher information S stored in the teacher information storage unit 422 by the position estimating unit 421 The accuracy of machine learning using (t, g, h | x, y) decreases.

Based on this point, the teacher information generation unit 423 calculates the temperature of the device, in this example, the temperature of the panel 10, and the teacher information S(t, g, h | x, y) stored in the teacher information storage unit 422. When the difference between the temperature of the device when all the information is generated and stored in the teacher information storage section 422 (in this example, the temperature of the panel 10) exceeds the set value, the device temperature determination section 45 displays the teacher information on the display screen of the panel 10. Prompts acquisition of information S(t, g, h | x, y).

Below, differences from the operation panel device according to Embodiment 1 will be mainly explained.
In the operation panel device according to the fifth embodiment, the position estimation operation and the calibration execution operation, that is, the teacher whose calibration result data is associated with position information (x, y) on the display screen of the panel 10; The operation of generating the information S(t, g, h | x, y) is the position estimation operation according to the flowchart shown in FIG. 7 and the calibration according to the flowchart shown in FIG. Since this is the same as the execution operation, the explanation will be omitted.

Therefore, the device temperature determination unit 45 generates the teacher information S (t, g, h | The difference between the temperature of the panel 10 at the time and the temperature of the panel at the set time when the position estimation operation is performed is compared, and if the difference between the two temperatures exceeds the difference set value, the teacher information S(t, g, The point of encouraging reacquisition of h|x, y) will be mainly explained.

The device temperature measuring section 46 detects the temperature of the device, in this example, the temperature of the panel 10.
The teacher information generation unit 423 generates the teacher information S (t, g, h | x, y) to be stored in the teacher information storage unit 422, and when the teacher information S (t, g, h | The temperature of the panel 10 detected by the device temperature measurement unit 46 is periodically acquired upon completion of the calibration execution operation and after a set time has elapsed after the completion of the calibration execution operation, and is provided to the device temperature determination unit 45 .

The device temperature determination section 45 stores the temperature of the panel 10 at the time of completion of the calibration execution operation from the teacher information generation section 423 as a set value.
The device temperature determination unit 45 uses the temperature of the panel 10 that is periodically acquired by the teacher information generation unit 423 as a comparison value, and compares the stored setting value and the comparison value.
If the difference between the set value and the comparison value is greater than or equal to the difference set value, the device temperature determination unit 45 generates temperature image information that displays an image that prompts the reacquisition of the teacher information S (t, g, h | x, y). , is applied to the panel 10 via the teacher information generation unit 423 and the control unit 50, and an image indicating the necessity of reacquiring the teacher information is displayed on the display screen of the panel 10.

Note that although the device temperature determination section 45 acquires the temperature of the panel 10 detected by the device temperature measurement section 46 via the teacher information generation section 423, it may directly acquire it from the device temperature measurement section 46.
Further, the device temperature determination section 45 may provide the temperature image information to the panel 10 via the control section 50 without going through the teacher information generation section 423.

When an image prompting re-acquisition of teacher information is displayed on the display screen of the panel 10, the user causes the operation panel device to acquire the teacher information.
The calibration execution operation in the operation panel device, that is, the teacher information S(t, g, h | x, y ) is the same as the operation described in Embodiment 1, and is executed according to the flowchart shown in FIG.

However, the teacher information S (t, g _{, h} | When stored in _the teacher information storage section 422, the _teacher information S(t, g, h|x , y).

Further, when the teacher information generation unit 423 completes the operation of re-acquiring the teacher information S (t, g, h | Upon completion of the re-acquisition operation, the teacher information generation unit 423 rewrites the temperature to the temperature of the panel 10 acquired from the device temperature measurement unit 46 and stores it as a new setting value.

As described above, the operation panel device according to the fifth embodiment has the same effects as the operation panel device according to the first embodiment.
Further, in the operation panel device according to the fifth embodiment, the device temperature determination unit 45 compares the temperature of the panel 10 with the set value of the temperature of the panel 10, and when the difference between both temperatures becomes equal to or larger than the difference set value, the temperature of the panel 10 is Since the elapsed time image information that causes the display screen to display an image that prompts the acquisition of teacher information S (t, g, h | x, y) is given to the panel 10, the first transparent antenna 2 ₁ to the Nth transparent antenna 2 The device temperature drift with respect to _N can be suppressed within the specified value, it is not easily affected by the temperature of the environment in which the operation panel device is used, and the finger position can be estimated using the teacher information S (t, g, h | x, y). .

Note that any one of the receiving sensitivity determining section 43 shown in the third embodiment, the elapsed time determining section 44 shown in the fourth embodiment, or the device temperature determining section 45 shown in the fifth embodiment may be replaced with the receiving sensitivity determining section 43 shown in the third embodiment. It may be applied to such an operation panel device.
Furthermore, it is possible to freely combine each embodiment, to modify any component of each embodiment, or to omit any component in each embodiment.

The operation panel device according to the present disclosure is suitable for a non-contact type operation panel device as a user interface used in various electronic devices having a touch display function, such as mobile terminals and portable devices.
Further, the present invention is suitable for operation panel devices used in factories and other environments where users with dirty fingers tend to get dirt on the display screen of the panel.

Reference Signs List 10 display panel, 20 transparent antenna array, 2 ₁₁ to 2 _mn , 2 ₁ to 2 _N first transparent antenna to Nth transparent antenna, 30 transmitting/receiving circuit section, 31 input/output switching section, 31 ₁ to 31 _N first 32 signal transmitter, 321 signal generator, 322 output destination selector, 33 reception information generator, 33 ₁ to 33 _N first signal receiver to Nth signal receiver. , 40 object position estimation unit, 41 reception information acquisition unit, 42 machine learning unit, 421 position estimation unit, 422 teacher information storage unit, 423 teacher information generation unit, 43 reception sensitivity determination unit, 44 elapsed time determination unit, 45 device temperature determination section, 50 control section;

Claims (14)

1. a display panel having a display screen;
   a plurality of transparent antennas two-dimensionally arranged on the display screen of the display panel;
   The plurality of transparent antennas are linked to the transparent antennas that have received the reflected waves based on the received signals from the plurality of transparent antennas that have received the reflected waves reflected from the object present on the display screen of the display panel. a reception information generation unit that generates reception information corresponding to the antenna;
   a storage unit that stores teacher information corresponding to the plurality of transparent antennas linked to position information on the display screen of the display panel;
   position estimation for estimating the position of the object on the display screen of the display panel by performing machine learning on the reception information generated by the reception information generation unit using teacher information stored in the storage unit; Department and
   An operation panel device comprising:
2. providing the display panel with instruction image information for displaying an instruction image that makes the object exist on the display screen of the display panel, and selecting the object that exists along the instruction image displayed on the display screen of the display panel; Position information on the display screen of the display panel based on the instruction image is included in reception information corresponding to the transparent antenna that received the reflected wave based on reception signals from the plurality of transparent antennas that received the reflected wave. The operation panel device according to claim 1, further comprising a teacher information generation unit that generates linked teacher information to be stored in the storage unit.
3. The display panel is provided with instruction image information for displaying an instruction image that causes the object to be present on the display screen of the display panel, and among the plurality of transparent antennas, the instruction image displayed on the display screen of the display panel is activating a plurality of transparent antennas present around the pointing position on the display screen of the display panel, and causing a reflection reflected from the object present along the instruction image displayed on the display screen of the display panel; Position information on the display screen of the display panel indicated by the instruction image is included in the reception information corresponding to the transparent antenna that received the reflected wave using the received signal from the activated plurality of transparent antennas that received the wave. The operation panel device according to claim 1, further comprising a teacher information generation unit that generates linked teacher information to be stored in the storage unit.
4. Provide instruction image information to the display panel to display an instruction image indicating a trajectory for moving the object on the display screen of the display panel, and move the object along the trajectory of the instruction image displayed on the display screen of the display panel. Based on the received signals from the plurality of transparent antennas that received the reflected waves reflected from the target object, the received information corresponding to the transparent antennas that received the reflected waves are displayed on the display screen of the display panel according to the instruction image. The operation panel device according to claim 1, further comprising a teacher information generation unit that generates teacher information stored in the storage unit and associated with position information at the storage unit.
5. The operation panel device according to any one of claims 1 to 4, wherein the machine learning by the position estimation unit uses a machine learning method of deep learning or Gaussian process regression.
6. a reception sensitivity determining unit that provides the display panel with reception sensitivity deterioration information that causes an image prompting the acquisition of teacher information to be displayed on the display screen when the reception sensitivity of the reception signals from the plurality of transparent antennas is below a set value; The operation panel device according to any one of claims 2 to 4.
7. The teacher information generation unit generates teacher information to be stored in the storage unit, and when the elapsed time from the time when the teacher information was stored in the storage unit exceeds a set time, displays an image on the display screen prompting the acquisition of teacher information. The operation panel device according to any one of claims 2 to 4, further comprising an elapsed time determination section that provides elapsed time image information to be displayed on the display panel.
8. When the difference between the temperature of the operation panel device and the temperature of the operation panel device at the time when the teacher information generation section generated the teacher information stored in the storage section exceeds a set value, the teacher information is displayed on the display screen. The operation panel device according to any one of claims 2 to 4, further comprising a device temperature determination section that provides the display panel with temperature image information that causes an image to be displayed that prompts the user to obtain the information.
9. The reception information generation unit generates reception signals from a plurality of transparent antennas arranged two-dimensionally on the display screen of the display panel that have received reflected waves reflected from objects present on the display screen of the display panel. and generating reception information corresponding to the plurality of transparent antennas linked to the transparent antenna that received the reflected wave, based on the received reception signal;
   A position estimating unit performs machine learning on the received information generated by the received information generating unit using teacher information stored in a storage unit, and calculates the position of the target on the display screen of the display panel. a step of estimating;
   An object position estimation method for estimating the position of an object on a display screen of an operation panel device.
10. The teacher information generation unit provides the display panel with instruction image information for displaying an instruction image that makes the object exist on the display screen of the display panel, and the teacher information generation unit provides instruction image information for displaying an instruction image that makes the object exist on the display screen of the display panel, Receiving signals from the plurality of transparent antennas that have received reflected waves from the existing target object, and adding the instruction image to reception information corresponding to the transparent antenna that received the reflected waves based on the received reception signals. generating teacher information stored in the storage unit, which is associated with position information on the display screen of the display panel;
    The object position estimation method according to claim 9, comprising:
11. Receive reflected waves based on received signals from a plurality of transparent antennas arranged two-dimensionally on the display screen of the display panel, which have received reflected waves reflected from objects present on the display screen of the display panel. a step of acquiring reception information corresponding to the plurality of transparent antennas linked to the transparent antenna;
    performing machine learning using teacher information on the acquired received information to estimate the position of the object on the display screen of the display panel;
    A program for estimating the position of an object on a display screen of an operation panel device, which causes a computer to execute this program.
12. providing the display panel with instruction image information for displaying an instruction image that makes the object exist on the display screen of the display panel, and selecting the object that exists along the instruction image displayed on the display screen of the display panel; Position information on the display screen of the display panel based on the instruction image is included in reception information corresponding to the transparent antenna that received the reflected wave based on reception signals from the plurality of transparent antennas that received the reflected wave. A procedure for generating linked teacher information,
    The program for estimating an object position according to claim 11.
13. Receive reflected waves based on received signals from a plurality of transparent antennas arranged two-dimensionally on the display screen of the display panel, which have received reflected waves reflected from objects present on the display screen of the display panel. a step of acquiring reception information corresponding to the plurality of transparent antennas linked to the transparent antenna;
    performing machine learning using teacher information on the acquired received information to estimate the position of the object on the display screen of the display panel;
    A recording medium storing a program for estimating the position of an object on the display screen of an operation panel device, which causes a computer to execute the program.
14. providing the display panel with instruction image information for displaying an instruction image that makes the object exist on the display screen of the display panel, and selecting the object that exists along the instruction image displayed on the display screen of the display panel; Position information on the display screen of the display panel based on the instruction image is included in reception information corresponding to the transparent antenna that received the reflected wave based on the reception signals from the plurality of transparent antennas that received the reflected wave. A procedure for generating linked teacher information,
    The recording medium according to claim 13, comprising:

Images