TW202411826A - Operation panel device, object position estimation method, object position estimation program, and recording medium - Google Patents

Abstract

An operation panel device comprises: a display panel (10), a plurality of transparent antennas (2 ₁ ) to (2 _N ), a reception information generating unit (330), a storage unit (422), and an object position estimating unit (40); the display panel (10) having a display screen; the plurality of transparent antennas (2 ₁ ) to (2 _N ) being two-dimensionally arranged on the display screen of the display panel; a reception information generating unit (33), wherein the plurality of transparent antennas (2 ₁ ) to (2 _N ) receive reflected waves reflected from an object existing on the display screen of the display panel (10); and the reception information generating unit (33) generates, based on reception signals from the plurality of transparent antennas (2 ₁ ) to (2 _N ), a reception information generating unit (33) which is additionally bound to the transparent antennas (2 ₁ ) to (2 _N ) that receive the reflected waves and corresponds to the plurality of transparent antennas (2 ₁ ) to (2 _N ) of received information; a storage unit (422) storing teacher information corresponding to a plurality of transparent antennas (2 ₁ ) to (2 _N ) and bound to the position information on the display screen of the display panel (10); and an object position estimation unit (40) performing machine learning using the teacher information stored in the storage unit (422) on the received information generated by the received information generation unit (330) to estimate the position of the object on the display screen of the display panel (10).

Description

Operation panel device, object position estimation method, object position estimation program, and recording medium

The present disclosure relates to an operation panel device having a function of estimating the position of an object on a display screen of a display panel.

In recent years, in various electronic devices, most of them use a touch operation panel as a user interface (User interface, hereinafter also referred to as UI). Moreover, Patent Document 1 proposes a sensing screen device using a non-contact operation panel as a UI. In the sensing screen device shown in Patent Document 1, the transparent antenna layer arranged directly above the display screen of the sensing screen includes: a first antenna unit and a plurality of second antenna units, the first antenna unit sends a sensing signal, and the plurality of second antenna units receive the reflected signal of the sensing signal. When the reflected signal of the sensing signal is received, the coordinates of the contact object on the plane are determined according to the power of the received reflected signal and the coordinates of the second antenna unit that received the reflected signal on the plane, wherein the aforementioned plane is the plane where the sensing screen is located. ＜Prior Technical Document＞ ＜Patent Document＞

<Patent Document 1> Japanese Patent Publication No. 2017-527146.

＜Problems that inventions are intended to solve＞

The sensing image device shown in Patent Document 1 estimates the position of the contact object by using the second antenna unit that receives the high-power reflection signal among a plurality of second antenna units that receive the reflection signal. Therefore, there is a problem that the estimation accuracy of the position of the contact object is poor.

This disclosure is to solve the above-mentioned problem. The purpose of this disclosure is to obtain an operation panel device that can estimate the position of an object on the display screen of the display panel with high accuracy. ＜Means for solving the problem＞

The operation panel device disclosed in the present invention comprises: a display panel, a plurality of transparent antennas, a receiving information generating unit, a storage unit and an object position estimating unit; the display panel has a display screen; a plurality of transparent antennas are two-dimensionally arranged on the display screen of the display panel; the receiving information generating unit, wherein the plurality of transparent antennas receive reflected waves reflected from objects existing on the display screen of the display panel, and the receiving information generating unit generates a received information based on the reflected waves from the plurality of transparent antennas. A receiving signal generating receiving information that is additionally bound to the transparent antenna that received the reflected wave and corresponds to a plurality of transparent antennas; a storage unit storing teacher information additionally bound to the position information on the display screen of the display panel and corresponding to a plurality of transparent antennas; and a position estimation unit that performs machine learning using the teacher information stored in the storage unit on the receiving information generated by the receiving information generating unit to estimate the position of the object on the display screen of the display panel. ＜Effect of the invention＞

According to the present disclosure, the object position estimation unit performs machine learning using teacher information stored in a storage unit on the reception information corresponding to a plurality of transparent antennas generated by the reception information generation unit to estimate the position of the object on the display screen of the display panel, thereby being able to estimate the position of the object on the display screen of the display panel with high accuracy.

[First embodiment] The operation panel device of the first embodiment is described using FIGS. 1 to 8. The operation panel device of the first embodiment is a non-contact operation panel with a calibration function, including: a display panel (hereinafter referred to as the panel) 10, a transparent antenna array 20, a transmitting and receiving circuit unit 30, an object position estimation unit 40, and a control unit 50. That is, the operation panel device of the first embodiment is a user interface (UI) that enables an object such as a finger or a stylus (hereinafter, the object is represented by a finger) to approach the panel 10 and operate the screen without physically touching the panel.

The operation panel device of the first embodiment has the following functions: a function of estimating the position of a finger on the display screen of the panel 10 by machine learning using teacher information for received information, i.e., a position estimation function, wherein the received information is based on received signals from a plurality of transparent antennas, and the plurality of transparent antennas constitute a transparent antenna array 20 that receives reflected waves from the finger. In addition, the operation panel device has a function of obtaining a plurality of teacher information, i.e., a calibration function, wherein the plurality of teacher information is respectively associated with each of the received information based on the received signals from the plurality of transparent antennas.

The panel 10 has a display screen. The transparent antenna array 20 is composed of a plurality of transparent antennas 2 11 to 2 _mn which are two _- dimensionally arranged and embedded in the entire interior of the panel 10 in order to improve the estimation accuracy of the finger position by using Multiple-Input Multiple-Output (MIMO).

As shown in FIG. 2 , the plurality of transparent antennas 2 ₁₁ to 2 _mn are two-dimensionally arranged in m rows (i.e., horizontal columns) and n columns (i.e., vertical columns) on the entire plane of the display screen of the panel 10. m and n are natural numbers, and at least one of them is a natural number greater than 2. For the convenience of explanation, the transparent antenna 2 ₁₁ is called the first transparent antenna 2 ₁ , and the transparent antenna 2 ₁₂ is called the second transparent antenna 2 ₂ , and the transparent antenna 2 _mn is called the Nth transparent antenna 2 _N . That is, the transparent antenna array 20 is composed of N transparent antennas from the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N. N is a natural number greater than 2. In addition, when it is not necessary to distinguish the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N , they are described as transparent antennas 2 _k (k=1 to N).

The transparent antenna _2k is a transmitting and receiving antenna that serves as both a transmitting antenna and a receiving antenna. In addition, the transparent antenna _2k may be a transmitting antenna and a receiving antenna that are separated individually. In addition, for N receiving antennas, the number of transmitting antennas may be less than the number of receiving antennas, for example, N/2. When the transparent antenna _2k functions as a transmitting antenna, it radiates a transmission wave composed of radio waves onto the display screen of the panel 10, that is, the space where the finger exists. The transmission wave is a high-frequency wave composed of a transmission signal of a high-frequency band such as the L band, S band, C band, or X band.

When the transparent antenna _2k functions as a receiving antenna, it receives the reflected wave reflected by the finger placed on the display screen of the panel 10 and outputs a received signal based on the reflected wave. The transparent antenna _2k is an antenna made of a transparent conductor such as a finely wired metal mesh, indium tin oxide film, or graphene, and is an antenna whose shape cannot be recognized by the naked eye.

The transmitting and receiving circuit unit 30 includes an input/output switching unit 31 , a signal transmitting unit 32 , and a receiving information generating unit 33 . The transmitting and receiving circuit section 30 selects the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N in sequence in each cycle and outputs a high-frequency wave transmission signal at the selected transparent antenna 2 _k . For each transmission wave of each selected transparent antenna 2 _k , the reflected wave reflected from the finger on the display screen of the panel 10 is received by the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N. The transmitting and receiving circuit section 30 receives the reception signal from the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N. Based on the received reception signal, the receiving information corresponding to the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N is generated and output. The transparent antenna 2 _k selected as the transmission antenna is additionally bound to the transparent antenna that received the reflected wave.

The input-output switching unit 31 transmits the transmission signal of the high frequency wave from the signal transmitting unit 32 to the transparent antenna 2 _k selected by the signal transmitting unit 32, and transmits the reception signal from the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N that received the reflected wave reflected from the finger to the reception information generating unit 33. The input-output switching unit 31 corresponds to the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N and has the first switching unit 31 ₁ to the Nth switching unit 31 _N. When it is not necessary to distinguish and explain the first switching unit 31 ₁ to the Nth switching unit 31 _N , they can be explained as switching units 31 _k (k=1 to N).

The switch section 31 _k is a known circulator with three ports widely used in the field of high frequency signals. The switch section 31 _k has an input port, an input/output port, and an output port. The input port of the switch section 31 _k is connected to the output port of the signal transmission section 32, the input/output port is connected to the corresponding transparent antenna 2 _k , and the output port is connected to the receiving information generation section 33. The switch section 31 _k is such that when transmitting, the input port is connected to the input/output port, and when receiving, the input/output port is connected to the output port.

The switch unit 31 _k connects the transparent antenna 2 _k to the reception information generating unit 33. The switch unit 31 _k selected by the signal transmitting unit 32 connects the corresponding transparent antenna 2 _k to the signal transmitting unit 32, and after outputting the transmission signal from the signal transmitting unit 32 to the transparent antenna 2 _k , connects the transparent antenna 2 _k to the reception information generating unit 33.

The signal transmission unit 32 selects the first switching unit 31 ₁ to the Nth switching unit 31 _N in sequence in each cycle and outputs a high-frequency transmission signal to the selected switching unit 31 _k . The selection order of the first switching unit 31 ₁ to the Nth switching unit 31 _N is the order of the first switching unit 31 ₁ , the second switching unit 31 ₂ , ..., the Nth switching unit 31 _N. In FIG. 2 , the order of selecting the transparent antenna is from the left end to the right end of the first row (row, i.e., horizontal row), from the left end to the right end of the second row, and from the left end to the right end of the mth row. In addition, the selection order from the first switch 31 ₁ to the Nth switch 31 _N may be the order of the Nth switch 31 _N , the (N-1)th switch 31 _N-1 , ..., the first switch 31 ₁ , etc., and the switching order is not limited.

The signal transmission unit 32 includes a signal generator 321 and an output destination selection unit 322. The signal generator 321 generates an up-modulated signal or a down-modulated signal according to a frequency modulated continuous wave (FMCW) method or a fast linear modulation (FCM) method, and outputs N transmission signals in each transmission cycle. The signal generator 321 repeatedly outputs N transmission signals in each transmission cycle c (c=1, 2, ...).

In the first embodiment, as shown in FIG. 3, the signal generator 321 generates up-modulated signals Tx1 to TxN whose frequencies change with time in each transmission cycle, up-converts the frequencies of the up-modulated signals Tx1 to TxN into signals of a high frequency band, and outputs the up-converted signals as transmission signals TX1 to TXN to the output destination selection unit 322 as high frequency bands, such as L band, S band, C band, X band, etc. The signal generator 321 generates the up-modulated signals Tx1 to TxN by a timing signal indicating a moment from the control unit 50.

The number N of transmission signals TX1-TXN in one transmission cycle is the same as the number N of the first transparent antenna ₂₁ to the Nth transparent antenna _2N . In the first embodiment, the first transmission signal TX1 to the Nth transmission signal TXN in one transmission cycle are additionally corresponding to the first transparent antenna ₂₁ to the Nth transparent antenna _2N . The order of generating the up-modulated signals Tx1-TxN by the signal generator 321 is the same as the order of selecting the first transparent antenna ₂₁ to the Nth transparent antenna _2N as the transmission antenna, that is, the order of the first signal Tx1, the second signal Tx2, ..., the Nth signal TxN.

In the case where it is not necessary to distinguish N transmission signals TX1 to TXN, they are described as transmission signals TXk (k=1 to N). In addition, the transmission signal TXk may be a signal based on a pulse signal instead of a signal whose frequency changes over time, such as an up-modulation signal or a down-modulation signal.

The output destination selection unit 322 obtains the transmission signal TXk from the signal generator 321 in synchronization with the time point signal indicating the time from the control unit 50, selects the switch unit _31k connected to the transparent antenna _2k corresponding to the obtained transmission signal TXk, and outputs the transmission signal TXk to the selected switch unit _31k . The output end of the output destination selection unit 322 is connected to the input ports of the first switch unit ₃₁₁ to the Nth switch unit _31N .

The reception information generating unit 33 is input with reception signals from the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N which receive the reflected wave reflected by the finger via the first switching unit 31 ₁ to the Nth switching unit 31 _N , wherein the finger is present on the display screen of the panel 10, and the reception information generating unit 33 generates and outputs reception information corresponding to the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N based on the input reception signals, wherein the transparent antenna 2 _k selected as the transmitting antenna is additionally bound to the transparent antenna 2 _k which receives the reflected wave.

The reception information generating unit 33 includes first signal receiving units 33 ₁ to 33 _N corresponding to the first transparent antenna 2 ₁ to the N-th transparent antenna 2 _N and corresponding to the first switching unit 31 ₁ to the N-th switching unit 31 _N. When it is not necessary to distinguish the first signal receiving unit 33 ₁ to the N-th signal receiving unit 33 _N , they are described as signal receiving units 33 _k (k=1 to N).

The input end of the signal receiving unit 33 _k is connected to the output port of the corresponding switching unit 31 _k . The signal receiving unit 33 _k receives the reception signal from the corresponding transparent antenna 2 _k via the corresponding switching unit 31 _k . The signal receiving unit 33 _k has: a down-converting unit and an analog/digital conversion unit. The down-converting unit down-converts the frequency of the high frequency band of the reception signal into the frequency of the intermediate frequency band. The analog/digital conversion unit converts the reception signal A/D (Analog/Digital) which is down-converted by the down-converting unit and is an analog signal into a reception signal which is a digital signal.

The signal receiving unit _33k receives a time signal indicating the time from the control unit 50, obtains information about the transparent antenna _2k selected as the transmitting antenna, the time point when the selected transparent antenna _2k radiates the transmitting wave, and the time point when the corresponding transparent antenna _2k receives the reflected wave from the finger and receives the receiving signal through the corresponding switching unit _31k , and obtains the frequency and intensity of the receiving signal received through the corresponding switching unit _31k .

Since the time when the selected transparent antenna _2k radiates the transmission wave is determined in the order of selection from the first transparent antenna ₂₁ to the Nth transparent antenna _2N , the signal receiving unit _33k inputs the time when the selected transparent antenna _2k radiates the transmission wave through the control unit 50. In addition, the time when the received signal is received is obtained by the signal receiving unit _33k based on the signal indicating the time from the control unit 50.

The signal receiving unit 33 _k generates receiving information S (t, g, h) based on the receiving signal received by the corresponding switching unit 31 _k and outputs it to the object position estimation unit 40, wherein the receiving information S (t, g, h) is digital data that binds the transparent antenna 2 _k selected as the transmission antenna and the transparent antenna 2 _k that receives the reflected wave. The receiving information S (t, g, h) includes information indicating the frequency and intensity of the receiving signal obtained from the transparent antenna 2 _k . The receiving information S (t, g, h) may also include changes in the spectrum shape of the receiving signal.

In the received information S (t, g, h), t is the reflected wave reception time, which indicates the sampling time when the sampling start time of each receiving antenna is set to 0 and is the digital data from (received) transparent antenna _2k ; g is the transmitting antenna identification information (for example: g = 1 to N), which identifies the digital data of the (transmitting) transparent antenna _2k selected as the transmitting antenna; h is the receiving antenna identification information (for example: h = 1 to N), which identifies the digital data of the transparent antenna _2k that received the reflected wave.

Assuming that the first transparent antenna 2 ₁ is selected as the transmitting antenna, the signal generator 321 outputs the transmitting signal TX1, and the first switching unit 31 ₁ is selected by the output destination selection unit 322 and the transmitting signal TX1 is input to the first transparent antenna 2 ₁ , and the first transparent antenna 2 ₁ radiates the transmitting wave composed of radio waves to the space on the display screen of the panel 10 (i.e., where the finger exists). In the k-th signal receiving unit 33 _k , after receiving the receiving signal from the corresponding transparent antenna 2 _k through the corresponding switching unit 31 _k , the receiving information S (t, 1, k) is generated, wherein the receiving signal from the aforementioned transparent antenna 2 _k is based on the received reflected wave reflected by the finger.

When the first transparent antenna ₂₁ is selected as the transmission antenna, the first signal receiving unit ₃₃₁ to the first signal receiving unit _33N will generate reception information S, so N reception information S (t, 1, h) will be generated for the reception information generating unit 33. In addition, in one transmission cycle, since the first transparent antenna ₂₁ to the Nth transparent antenna _2N are sequentially selected as the transmission antenna, (N×N) reception information S (t, g, h) will be generated for the reception information generating unit 33 and output to the object position estimating unit 40.

The object position estimation unit 40 has two functions. The first function is to perform machine learning using teacher information for the received information S (t, g, h) generated by the received information generation unit 33, in this example, (N×N) received information S (t, g, h) generated in one transmission cycle, to estimate the position of the finger on the display screen of the panel 10. In addition, as is generally known, after machine learning is performed using teacher information to establish a learning model, the learning model is used to estimate the position of the finger on the display screen of the panel 10.

The second function is to display an indication image IP for indicating that a finger is present on the display screen of the panel 10, and the finger is located at a set position along the displayed indication image IP. Based on the reception signals from the first transparent antenna ₂₁ to the Nth transparent antenna _2N that received the reflected wave reflected by the finger, the reception information corresponding to the transparent antenna that received the reflected wave (that is, the (N×N) reception information S (t, g, h) in each transmission cycle generated by the reception information generating unit 33) is used as calibration information to generate calibration result data additionally bound to the position information (x, y) on the display screen of the panel 10 obtained from the indication image IP, that is, the teacher information S (t, g, h | x, y), that is, the calibration function. x, y are the x coordinate and y coordinate on the display screen when the finger where the reflected wave is received exists at the set position along the indication image IP. In addition, the coordinates (z coordinates) in the vertical direction relative to the display screen of the panel 10 may also be added as position information.

The teacher information S (t, g, h | x, y) is information generated by pairing the position information (x, y) with the total received information S (t, g, h), wherein the position information (x, y) is the position on the display screen of the panel 10 obtained by the indication image IP, and the received information S (t, g, h) is obtained based on each received signal from the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N that receives the reflected wave of the finger located at this position, and is the total received information S (t, g, h) generated by the first signal receiving unit 33 ₁ to the first signal receiving unit 33 _N.

The object position estimation unit 40 includes a reception information acquisition unit 41 and a machine learning unit 42. The reception information acquisition unit 41 acquires reception information S (t, g, h) generated by each of the first signal receiving unit 33 ₁ to the first signal receiving unit 33 _N. The reception information acquisition unit 41 acquires (N×N) reception information S (t, g, _h ) generated by the reception information generation unit 33 composed of the first signal receiving unit 33 ₁ to the first signal receiving unit 33 N in each transmission cycle c, and provides the machine learning unit 42.

The machine learning unit 42 has the above-mentioned first function and second function. The machine learning unit 42 includes: a position estimation unit 421, a teacher information storage unit 422, and a teacher information generation unit 423. The position estimation unit 421 performs machine learning on the received information S (t, g, h) obtained by the received information acquisition unit 41, in this example, for the (N×N) received information S (t, g, h) in each sending cycle c, wherein the machine learning uses the plurality of teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 and each of which is bound to the plurality of position information (x, y) of the display screen of the panel 10, to estimate the position (X, Y) of the finger on the display screen of the panel 10, and outputs the estimated position of the finger.

The machine learning performed by the position estimation unit 421 is machine learning performed using either deep learning or Gaussian process regression. Other machine learning frameworks may be used instead of deep learning and Gaussian process regression machine learning methods. When deep learning is used as the machine learning method, the weight parameters connecting the layers are equivalent to teacher information.

For example, in a machine learning framework using deep learning, the teacher information S(t, g, h | x, y) obtained at all positions (x, y) is used as the learning data input, and the real finger position (x, y) is used as the output to learn the deep learning model. All weight parameters in the deep learning model are adjusted by error back propagation method, etc., so that the deep learning model outputs a value close to the real finger position (x, y). All these weight parameters are equivalent to the teacher information in the deep learning model.

When using Gaussian process regression as a machine learning method, the Gram matrix parameters are equivalent to the teacher information. For example, in Gaussian process regression, the approximation of all pairwise pairs between the teacher information S(t, g, h | x, y) obtained at all positions (x, y) is calculated using a predetermined kernel function, and a Gram matrix of M columns × M rows is generated, where the matrix elements are the outputs of these kernel functions. Here, M is the number of all learning positions (x, y). The predetermined kernel function is, for example, a Gaussian kernel. For the received information S (t, g, h | X, Y) in the case of an unknown finger position (X, Y), the unknown finger position (X, Y) is estimated by performing matrix operations using this Gram matrix. This method is called the finger position estimation method using Gaussian process regression. This Gram matrix is equivalent to the teacher information. When using other machine learning frameworks with teachers, the parameters of this machine learning model with teachers are equivalent to the teacher information.

The teacher information generating unit 423 coordinates the control panel 10 and the transmitting and receiving circuit unit 30 by the control unit 50 when obtaining the calibration operation of a plurality of teacher information. The teacher information generating unit 423 displays the instruction image IP on the display screen of the panel 10 through the control unit 50. The instruction image IP is an image used to give instructions to the user, and the instruction is to slowly trace the slowly moving front end of the finger.

As shown in FIG. 4, the indication image IP is a meandering trace image that slowly traces the following trace: for example, from the upper left end to the upper right end of the display screen of the panel 10, it descends a section, from the right end to the left end, and after going back and forth several times, it ends from the lower right end to the lower left end. As shown in FIG. 4, the trace line as the indication image IP is a trace line that is slowly moved by the front end on the display screen of the panel 10 and the moved part becomes a dark trace line IPb. The light trace line IPa is a trace line that indicates that the dark trace line IPb is about to extend. The trace line as the indication image IP indicates that the user is instructed to slowly trace the movement of the front end on the display screen of the panel 10 with a finger. The time required for the front end to move on the light trace line IPa is several tens of seconds.

The teacher information generating unit 423 obtains the (N×N) received information S (t, g, h) in each transmission cycle c generated by the received information generating unit 33 as calibration information at each of the plurality of setting positions P _d (d=1 to D) located at a plurality of places (equally spaced in this example) in the drawing line of the instruction image IP (i.e., the first setting position P ₁ to the D-th setting position P _D ). D is a natural number greater than 2. That is, at each setting position P _d , the teacher information generating unit 423 inputs the transmission signal from the signal transmitting unit 32 to the selected transparent antenna 2 _k via the switching unit 31 _k , and radiates the transmission wave from the selected transparent antenna 2 _k .

The teacher information generating unit 423 obtains the receiving information corresponding to the transparent antenna that received the reflected wave through the receiving information acquiring unit 41, that is, the receiving information S (t, g, h) generated by the receiving information generating unit 33, wherein the receiving information is based on the receiving signals from the first transparent antenna ₂₁ to the Nth transparent antenna _2N , and the first transparent antenna ₂₁ to the Nth transparent antenna _2N receive the reflected wave reflected by the finger for the transmission wave from the selected transparent antenna _2k . The teacher information generating unit 423 obtains the received information S (t, g, h) generated by the received information generating unit 33 via the received information acquiring unit 41 at each setting position P _d , wherein each setting position P _d is the setting position P _d of the drawing line where the front end portion of the drawing line as the indication image IP is located when the front end portion of the drawing line is drawn slowly from the beginning (upper left end) to the end (lower left end) to draw a winding trajectory.

At each setting position P _d , in one transmission cycle c, the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N are sequentially selected as transmission antennas, and the teacher information generating unit 423 obtains (N×N) pieces of received information S (t, g, h) from the received information generating unit 33 via the received information acquiring unit 41. Thus, until the finger on the display screen of the panel 10 traces the front end of the drawing line drawn as the instruction image IP and completes the drawing line, at all positions from the first setting position P ₁ to the Dth setting position _PD , the teacher information generating unit 423 obtains (N×N) pieces of received information S (t, g, h) from the received information generating unit 33 via the received information acquiring unit 41.

At each setting position P _d , the teacher information generating unit 423 generates teacher information S (t, g, h | x d , y _d ) as calibration result data, wherein the calibration result data is the position information (x _d , y d ₎ of the front end of the indication image IP for sequentially selecting the setting position P _d of the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N as the transmitting antenna on the display screen of the panel 10, and is matched with the (N× _N ) receiving information S (t, g, h) from the information generating unit 33, and the teacher information S (t, g, h | x _d , y _d ) is stored in the teacher information storage unit 422.

That is, the teacher information generating unit 423 generates teacher information S (t, g, h | x, y) which is bound to the position information (x _d , y _d ) (d = 1 to D) of the front end of the indication image IP, that is, the position information (x _d , y _d ) obtained from the indication image IP on the display screen of the panel 10, at each of the positions set over the entire area of the indication image IP on the display screen of the panel 10, that is, the first setting position P ₁ to the D-th setting position _PD . In addition, the position information of the front end can be expressed by the coordinates of the x-axis and the y-axis on the display screen of the panel 10. In addition, the coordinates of the z-axis in the vertical direction with respect to the display screen of the panel 10, that is, with respect to the xy plane, can also be added as the position information of the front end.

Next, the specific structure of the object position estimation unit 40 is described. As a first example, the reception information acquisition unit 41 and the machine learning unit 42 are each implemented by dedicated hardware composed of a reception information acquisition circuit and a machine learning circuit as shown in FIG. 5. The reception information acquisition circuit corresponds to the reception information acquisition unit 41, and the machine learning circuit corresponds to the machine learning unit 42.

The received information acquisition circuit is an interface circuit that transmits the received information S (t, g, h) from the received information generation unit 33 to the machine learning circuit, and a generally known interface circuit can be used. The machine learning circuit includes: a position estimation circuit, a memory, and a teacher information generation circuit; the position estimation circuit constitutes a position estimation unit 421; the memory serves as a teacher information storage unit 422; and the teacher information generation circuit constitutes a teacher information generation unit 423. The position estimation circuit and the teacher information generation circuit can each be implemented by, for example, a single circuit, a complex circuit, a programmable processor, a parallel programmable processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof.

As a second example, the receiving information acquisition unit 41 and the machine learning unit 42 can be implemented by a computer with a program incorporated in software, firmware, or a combination of software and firmware as shown in FIG. 6. FIG. 6 shows a representative example of a computer, which includes: a central processing unit (Central Processing Unit, also known as CPU), an input interface, a random access memory (Random Access Memory, also known as RAM), a read-only memory (Read Only Memory, also known as ROM), and an output interface. The CPU controls and manages: an input interface, RAM, ROM, and an output interface.

The CPU functions as the position estimation unit 421 and the teacher information generation unit 423 together with the ROM and RAM. The ROM stores a position estimation program and a teacher information generation program. The position estimation program executes the position estimation processing program required for the position estimation unit 421 function, and the teacher information generation program is a calibration execution program that executes the processing program for generating teacher information required for the teacher information generation unit 423 function. ROM is a recording medium, which is a digital video disc (Digital Versatile Disc, also known as DVD), a compact disc (Compact Disc, also known as CD), a hard disk drive (Hard Disk Drive), or a universal serial bus (Universal Serial Bus, also known as USB) memory. RAM functions as the teacher information storage unit 422, and also plays the role of temporarily storing the program stored in ROM when the CPU executes the program stored in ROM.

The input interface part corresponds to the receiving information acquisition part 41 for acquiring the receiving information S (t, g, h) from the receiving information generation part 33, and when the computer performs the calibration execution action, the position information (x, y) of the front end in the display screen of the panel 10 is acquired based on the instruction information. The output interface part outputs the estimated position of the finger as the estimation result when the computer performs the position estimation action, wherein the estimated position of the finger is obtained by the CPU, ROM and RAM functioning as the position estimation part 421; when the computer performs the calibration execution action, the instruction image information is output by the CPU, ROM and RAM functioning as the teacher information generation part 423, wherein the instruction image information is the instruction image IP displayed on the display screen of the panel 10.

In addition, although the computer uses a CPU, it can also be hardware that executes programs of a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, or a digital signal processor (also referred to as a DSP). In addition, not limited to the first and second examples, some of the components may be implemented by dedicated hardware, and the remaining components may be implemented by software or firmware, etc.

Next, the operation of the operation panel device of the first embodiment is described. First, the position estimation operation of one transmission cycle c (c=1) is described using FIG. 7. Although multiple transmission cycles c are implemented, since the position estimation operation is the same, only one transmission cycle c (c=1) is described. In step ST11, the transparent antenna _2k selected as the transmission antenna is initially set to the first transmission antenna ₂₁ (k=1). In step ST12, in order to radiate the transmission wave from the first transparent antenna ₂₁ , the output destination selection unit 322 selects the first switching unit ₃₁₁ and outputs the first transmission signal TX1 from the signal generator 321 to the first transparent antenna ₂₁ .

The first transparent antenna ₂₁ radiates the transmission wave to the space where the finger exists. The transmission wave radiated by the first transparent antenna ₂₁ is reflected by the finger, and the first transparent antenna ₂₁ to the Nth transparent antenna _2N receive the reflected wave from the finger (step ST13). The reception signals from the reflected waves received by the first transparent antenna ₂₁ to the Nth transparent antenna _2N are input to the first signal receiving unit ₃₃₁ to the first signal receiving unit _33N through the first switching unit ₃₁₁ to the Nth switching unit _31N .

The first signal receiving unit 33 ₁ to the first signal receiving unit 33 _N respectively perform reception processing on each input reception signal, and generate reception information S (t, 1, h (1 to N)) based on the received reception signal for: the transparent antenna selected as the transmission antenna is the first transparent antenna 2 ₁ , the moment when the first transparent antenna 2 ₁ radiates the transmission wave, and the moment when the corresponding transparent antenna 2 _k receives the reflected wave from the finger (step ST14).

When the first signal receiving unit 33 ₁ to the first signal receiving unit 33 _N respectively generate the reception information S (t, 1, h (1 to N)) when the first transparent antenna 2 ₁ is selected as the transmission antenna, the process proceeds to step ST15, the transparent antenna 2 _k selected as the transmission antenna is set as the second transmission antenna 2 ₂ (k=2) and the process returns to step ST12. In step ST12, in order to make the transmission wave radiate from the second transmission antenna 2 ₂ (k=2) as the transmission antenna, the output destination selection unit 322 selects the second switching unit 31 ₂ and outputs the second transmission signal TX2 from the signal generator 321 to the second transparent antenna 2 ₂ .

The second transparent antenna ₂₂ radiates the transmission wave to the space where the finger exists. The transmission wave radiated by the second transparent antenna ₂₂ is reflected by the finger, and the first transparent antenna ₂₁ to the Nth transparent antenna _2N receive the reflected wave from the finger (step ST13). The reception signals from the reflected waves received by the first transparent antenna ₂₁ to the Nth transparent antenna _2N are input to the first signal receiving unit ₃₃₁ to the first signal receiving unit _33N through the first switching unit ₃₁₁ to the Nth switching unit _31N .

The first signal receiving unit 33 ₁ to the first signal receiving unit 33 _N respectively perform reception processing on each input reception signal, and generate reception information S (t, 2, h (1~N)) based on the received reception signal for: the transparent antenna selected as the transmission antenna is the second transparent antenna 2 ₂ , the moment when the second transparent antenna 2 ₂ radiates the transmission wave, and the moment when the corresponding transparent antenna 2 _k receives the reflected wave from the finger (step ST14).

In this way, until the Nth transparent antenna _2N is selected as the transmitting antenna, the transparent antenna _2k is selected as the transmitting antenna in sequence, and the loop from step ST12 to step ST15 is repeated, and (N×N) receiving information S (t, g, h) is generated by the first signal receiving unit ₃₃₁ to the first signal receiving unit _33N .

In step ST14, the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N which are two-dimensionally arranged on the display screen of the panel 10 receive reflected waves reflected from a finger existing on the display screen of the panel 10, and the reception information generating unit 33 is input with reception signals from the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N. Based on the input reception signals, reception information S (t, g, h) which is additionally bound to the transparent antenna 2 _k which receives the reflected waves and corresponds to the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N is generated.

In step ST15, if the Nth transparent antenna _2N is selected as the transmitting antenna, the process proceeds to step ST16. In step ST16, the reception information acquisition unit 41 acquires (N×N) reception information S (t, g, h) from the reception information generation unit 33, and provides the (N×N) reception information S (t, g, h) to the position estimation unit 421.

The position estimation unit 421 performs machine learning on the (N×N) received information S (t, g, h), wherein the machine learning uses a plurality of teacher information S (t, g, h|x, y) stored in the teacher information storage unit 422, and the plurality of teacher information S (t, g, h|x, y) is respectively bound to a plurality of position information (x, y) on the display screen; the position estimation unit 421 estimates the position (X, Y) of the finger on the display screen and outputs the estimated position of the finger. In addition, the subsequent estimation of the finger position is performed using the learning model generated by the result of machine learning.

In step ST16, the position estimation unit 421 in the object position estimation unit 40 performs machine learning on the (N×N) received information S (t, g, h) generated by the received information generation unit 33 using the teacher information S (t, g, h|x, y) stored in the teacher information storage unit 422, and estimates the position (X, Y) of the finger on the display screen of the panel 10.

In addition, when the received information acquisition unit 41 and the machine learning unit 42 are realized by a computer, a program for executing step ST16 is stored in a ROM, and the CPU executes the program stored in the ROM. The program stored in the ROM is a program for estimating the position of an object on a display screen of an operation panel device, including: a program for acquiring reception information S (t, g, h), wherein a first transparent antenna 2 ₁ to an Nth transparent antenna 2 _N two-dimensionally arranged on the display screen of the panel 10 receive a reflected wave reflected by a finger existing on the display screen of the panel 10, and based on the reception signals from the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N , the reception information S (t, g, h) additionally bound to the transparent antenna 2 _k that received the reflected wave and corresponding to the first transparent antenna 2 ₁ to the Nth transparent antenna 2 _N is generated; and a program for estimating the position, performing machine learning on the acquired reception information S (t, g, h) using teacher information S (t, g, h|x, y) to estimate the position (x, y) of the finger on the display screen of the panel 10.

Next, FIG. 8 is used to illustrate the calibration execution action in the operation panel device of the first embodiment, that is, the action of generating calibration result data bound to the position information (x, y) on the display screen of the panel 10 as the teacher information S (t, g, h | x, y). The teacher information generating unit 423 coordinates the control panel 10 and the transmitting and receiving circuit unit 30 through the control unit 50.

In step ST21, the teacher information generating unit 423 outputs the instruction image information of the display instruction image IP to the panel 10 via the control unit 50. As shown in FIG. 4, the panel 10 displays a meandering light drawing line IPa on the display screen, and the front end moves slowly along the light drawing line IPa, and the moved part becomes the dark drawing line IPb, thereby displaying the instruction image IP, indicating that the user should use his finger to follow the movement of the front end of the dark drawing line IPb and slowly trace it (step ST22).

When the front end of the deep drawing line IPb in the instruction image IP reaches the first setting position _P1 , the teacher information generating unit 423 obtains the position information ( _x1 , _y1 ) of the front end of the instruction image IP located at the first setting position _P1 (step ST23). At the same time, the teacher information generating unit 423 obtains (N×N) received information S (t, g, h) generated by the received information generating unit 33 through the received information acquiring unit 41. The aforementioned (N×N) received information S (t, g, h) is generated by the received information generating unit 33 based on the received signals received by the first transparent antenna ₂₁ to the Nth transparent antenna _2N when the first transparent antenna ₂₁ to the Nth transparent antenna _2N are each used as a transmitting antenna at the first setting position _P1 (step ST24).

In step ST24, the (N×N) pieces of received information S (t, g, h) obtained by the teacher information generating unit 423 are the same as the (N×N) pieces of received information S (t, g, h) generated by the received information generating unit 33 in the position estimation operation through steps ST11 to ST15. That is, although detailed description is omitted, at the first setting position _P1 , the first transparent antenna ₂₁ to the Nth transparent antenna _2N are all selected as the transmitting antenna in sequence, and the loop of steps ST12 to ST15 is repeated, and the teacher information generating unit 423 obtains the (N×N) pieces of received information S (t, g, h) generated by the first signal receiving unit ₃₃₁ to the first signal receiving unit _33N .

In step ST25, the teacher information generating unit 423 generates the teacher information S (t, g, h | x ₁ , y ₁ ) of the first setting position P ₁ by pairing the acquired position information (x 1 , y ₁ ) of the first setting position P ₁ with the acquired (N×N) receiving information S (t, g, h) of the first setting position P 1. The teacher information generating unit 423 stores the generated teacher information S (t, g, h | _{x 1 ,} y ₁ ) of the first setting position P ₁ in the teacher information storing unit 422 (step ST26), and the _process proceeds to step ST27.

Next, when the front end of the deep drawing line IPb of the indication image IP reaches the second setting position _P2 , the process returns to step ST23, and the teacher information generating unit 423 obtains the position information ( _x2 , _y2 ) of the front end of the indication image _IP at the second setting position _P2 . Thereafter, in step ST24, the teacher information generating unit 423, whose receiving information generating unit 33 generates (N×N) receiving information S by using the receiving signals received by the first transparent antenna ₂₁ to the Nth transparent antenna _2N when the first transparent antenna ₂₁ to the Nth transparent antenna _2N are each used as a transmitting antenna at the second setting position P2, and the teacher information generating unit 423 obtains the (N×N) receiving information S (t, g, h) generated by the receiving information generating unit 33.

In step ST25, the teacher information generating unit 423 pairs the acquired position information ( _x2 , _y2 ) of the second setting position _P2 with the acquired (N×N) receiving information S (t, g, h) of the second setting position _P2 to generate the teacher information S (t, g, h | _x2 , _y2 ) of the second setting position _P2 . In step ST26, the teacher information generating unit 423 stores the generated teacher information S (t, g, h | _x2 , _y2 ) of the second setting position _P2 in the teacher information storage unit 422 and proceeds to step ST27.

As described above, until the front end of the deep drawing line IPb of the indication image IP reaches the D-th setting position _PD , the loop from step ST23 to step ST27 is repeated in the setting position order from the first setting position _P1 to the D-th setting position _PD , and the teacher information S (t, g, h | x _d , y _d ) of each setting position P _d is stored in the teacher information storage unit 422. In step ST27, when d of the setting position P _d becomes D, that is, the teacher information S (t, g, h | x, y) is generated for all the positions set over the entire area of the indication image IP on the display screen of the panel 10, that is, the first setting position _P1 to the D-th setting position _PD , and is stored in the teacher information storage unit 422, the operation of generating the teacher information is completed.

Steps ST21 to ST27 are as follows: the teacher information generating unit 423 gives the panel 10 the indication image information, the indication image information displays the indication image IP on the display screen of the panel 10 so that the finger exists, the finger exists along the indication image IP displayed on the display screen of the panel 10, the first transparent antenna ₂₁ to the Nth transparent antenna _2N receive the reflected wave reflected by the finger, and the teacher information generating unit 423 receives the reflected wave from the first transparent antenna ₂₁ to the Nth transparent antenna 2 _N 's receiving signal generates teacher information S (t, g, h | x, y) which is additionally bound to the position information (x, y) on the display screen of the panel 10 indicated by the indication image IP based on the received receiving signal using the receiving information S (t, g, h) corresponding to the transparent antenna that received the reflected wave. The teacher information S (t, g, h | x, y) is stored in the teacher information storage unit 422.

In addition, when the received information acquisition unit 41 and the machine learning unit 42 are implemented by a computer, a program for executing the processing of steps ST21 to ST27 is stored in the ROM, and the CPU executes the program stored in the ROM. The program stored in the ROM is a teacher information generation program for estimating the position of an object on the display screen of the operation panel device, and includes the following procedures: providing the panel 10 with indication image information, the indication image information displaying an indication image IP on the display screen of the panel 10 showing that a finger exists, and a finger exists along the indication image IP displayed on the display screen of the panel 10, based on the reception signals from the first transparent antenna ₂₁ to the Nth transparent antenna _2N that receive the reflected wave reflected by the finger, using the reception information S (t, g, h) corresponding to the transparent antenna that receives the reflected wave, to generate teacher information S (t, g, h | x, y) additionally bound to the position information (x, y) on the display screen of the panel 10 obtained from the indication image IP.

As described above, the operation panel device of the first embodiment includes: the first transparent antenna ₂₁ to the Nth transparent antenna _2N , which are two-dimensionally arranged on the display screen of the panel 10; the reception information generating unit 33, based on the reception signal from the first transparent antenna ₂₁ to the Nth transparent antenna _2N that receives the reflected wave reflected by the finger on the display screen of the panel 10, generates the reception information generating unit 33 which is additionally bound to the transparent antenna that receives the reflected wave and corresponds to the first transparent antenna ₂₁ to the Nth transparent antenna 2N. _N 's received information S (t, g, h); the object position estimation unit 40 performs machine learning on the received information S (t, g, h) generated by the received information generating unit 33 using the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 to estimate the position of the finger on the display screen of the panel 10; thereby improving the accuracy of the finger position estimation when the finger is not in contact with the display screen of the panel 10.

In the operation panel device of the first embodiment, the teacher information generating unit 423 gives the panel 10 the indication image information, and the indication image information shows that the finger is present in the indication image IP on the display screen of the panel 10. The finger is present along the indication image IP shown on the display screen of the panel 10. The first transparent antenna ₂₁ to the Nth transparent antenna _2N receive the reflected wave reflected by the finger. Based on the information from the first transparent antenna ₂₁ to the Nth transparent antenna 2 The received signal of _N is additionally bound to the position information (x, y) on the display screen of the panel 10 obtained by the indication image IP, and the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 is generated. Therefore, it does not take a lot of time to generate and obtain the teacher information S (t, g, h | x, y), and it can be easily and easily realized. In addition, even if cracks or dirt are attached to the display screen of the panel 10, the teacher information S (t, g, h | x, y) can still be easily obtained again, preventing the accuracy of finger position estimation from deteriorating.

[Second embodiment] The operation panel device of the second embodiment is described with reference to FIG. 9. The teacher information generating unit 423 of the operation panel device of the first embodiment generates teacher information S ( _{t, g, h | x,} y) obtained according to the received information S (t, g, h) corresponding to the transparent antenna that received the reflected wave, based on the received signals from all of the first transparent antenna 21 to the Nth transparent antenna _2N that received the reflected wave reflected from the finger.

In contrast, the operating panel device of the second embodiment generates teacher information S ( _{t, g} , h | x, y) using the received information S (t, g, h) corresponding to the transparent antenna that received the reflected wave, based on the received signals from the plurality of antennas from the first transparent antenna 2 1 to the Nth transparent antenna 2 _{N (} here, the plurality of antennas exist around the position on the display screen of the panel 10 indicated by the indication image IP displayed on the display screen of the panel 10 and are activated). The operating panel device of the second embodiment is different from the operating panel device of the first embodiment in this point, and the other points are the same. In addition, in FIG. 9, the same symbols as those added in FIGS. 1 to 8 represent the same or equivalent parts.

The following description will focus on the differences between the operation panel device of the first embodiment. In the operation panel device of the second embodiment, the position estimation operation is the same as the position estimation operation of the operation panel device of the first embodiment, so its description is omitted. Therefore, the calibration execution operation, that is, the operation of generating calibration result data bound to the position information (x, y) on the display screen of the panel 10 as the teacher information S (t, g, h | x, y), will be described with reference to FIG. 8.

The operation panel device of the second embodiment is the same as the steps ST21 to ST23 in the operation panel device of the first embodiment. The teacher information generating unit 423 outputs the instruction image information for displaying the instruction image IP on the display screen of the panel 10 via the control unit 50 to the panel 10 (step ST21). As shown in FIG. 9, a light drawing line IPa with a meandering trajectory is displayed on the display screen, and the front end portion moves slowly along the light drawing line IPa, and the moved portion becomes a dark drawing line IPb, thereby displaying the instruction image IP, indicating that the user is instructed to trace the movement of the front end portion of the dark drawing line IPb with a finger and trace it slowly (step ST22).

Then, when the front end of the deep drawing line IPb in the instruction image IP reaches the first setting position _P1 , the teacher information generating unit 423 obtains the position information ( _x1 , _y1 ) of the front end of the instruction image IP located at the first setting position _P1 (step ST23). By the instruction from the teacher information generating unit 423, a plurality of transparent antennas (for example, 4 transparent antennas in the surroundings in this example) existing around the first setting position _P1 among the first transparent antenna ₂₁ to the Nth transparent antenna _2N are activated, that is, they are turned on. For example, in FIG. 9, as shown by the solid line frame, at the setting position _Pd , four transparent antennas, namely, transparent antenna ₂₁₂ , transparent antenna ₂₁₃ , transparent antenna ₂₂₂ , and transparent antenna ₂₂₃ , which surround the setting position _Pd, are activated.

Among the first transparent antenna ₂₁ to the Nth transparent antenna _2N , the transparent antennas other than the four activated transparent antennas are inactive, i.e., in an off state. This is indicated by a dashed frame in FIG. 9. In addition, in this example, the activation and deactivation of the transparent antenna are not limited to turning the transparent antenna on and off, but also include keeping the transparent antenna in an on state and turning the signal receiving unit corresponding to the transparent antenna on and off.

At the first setting position _P1 , corresponding to step ST24 of the operation panel device of the first embodiment, when the plurality of activated transparent antennas among the first transparent antenna ₂₁ to the Nth transparent antenna _2N are each used as a transmitting antenna, the reception information generating unit 33 generates, for example, (4×4) reception information S (t, g, h) by the reception signals received by the plurality of activated transparent antennas, and the teacher information generating unit 423 obtains the above-mentioned reception information S (t, g, h) via the reception information obtaining unit 41.

Thereafter, similar to step ST25 and step ST26 of the operation panel device of the first embodiment, the teacher information generating unit 423 generates teacher information S (t, g, h | x ₁ , y ₁ ) of the first setting position P ₁ by pairing the acquired position information (x ₁ , y 1 ) of the first setting position P ₁ with, for example, (4× ₄ ) received information S (t, g, h) of the first setting position P ₁ , and stores the generated teacher information S (t, g, h | x ₁ , y ₁ ) of the first setting position P ₁ in the teacher information storage unit 422 (step ST26), and proceeds to step ST27.

Next, when the front end of the deep drawing line IPb of the indication image IP reaches the second setting position _P2 , the teacher information generating unit 423 obtains the position information ( _x2 , _y2 ) of the front end of the indication image IP at the second setting position _P2 , and generates the teacher information S (t, g, h|x2, y2) of the _second setting position _P2 in the same way as the teacher information S (t, g, h| _x1 , _y1 ) of the first setting position _P1 , and stores _it in the teacher information storage unit 422. That is, a plurality of transparent antennas existing around the second setting position P ₂ are activated, and reception information S (t, g, h) based on reception signals from the activated transparent antennas is paired with the position information (x ₂ , y ₂ ) of the second setting position P ₂ to generate teacher information S (t, g, h | x ₂ , y ₂ ) of the second setting position P ₂ .

In this way, until the front end of the deep drawing line IPb of the indication image IP reaches the D-th setting position _PD , the teacher information S (t, g, h | x _d , y d ) of each setting position P _d is repeatedly generated in accordance with the setting position sequence from the first setting position P ₁ to the D-th setting position PD, corresponding to the transparent antenna around the setting position P _d , and stored in the teacher information storage unit 422. When the teacher information S (t, g, h | _x , _y ) is generated for all the positions set over the entire area of the indication image IP on the display screen of the panel 10 (i.e., the first setting position P ₁ to the D-th setting position _PD ) and stored in the teacher information storage unit 422, the operation of generating the teacher information is completed.

In the operation panel device of the second embodiment, the steps of generating teacher information S (t, g, h | x, y) are as follows: the teacher information generating unit 423 provides the panel 10 with indication image information, the indication image information displays the indication image IP on the display screen of the panel 10 so that the finger exists along the indication image IP displayed on the display screen of the panel 10, and the teacher information generating unit 423 receives the teacher information from the activated plurality of antennas (the plurality of antennas are the first transparent antenna ₂₁ to the Nth transparent antenna 2 The receiving signal is received from a plurality of antennas _N that receive the reflected wave reflected by the finger and exist around the position on the display screen of the panel 10 indicated by the indication image IP displayed on the display screen of the panel 10, and based on the received receiving signal, the receiving information S (t, g, h) corresponding to the transparent antenna that received the reflected wave is additionally bound to the position information (x, y) on the display screen of the panel 10 indicated by the indication image IP to generate teacher information S (t, g, h | x, y), and the teacher information S (t, g, h | x, y) is stored in the teacher information storage unit 422.

In addition, when the receiving information acquisition unit 41 and the machine learning unit 42 are implemented by a computer, a program for executing the processing of generating teacher information S (t, g, h | x, y) is stored in the ROM, wherein the teacher information S (t, g, h | x, y) is stored in the teacher information storage unit 422; the CPU executes the program stored in the ROM. The program stored in the ROM is a teacher information generation program for estimating the position of an object on the display screen of the operation panel device, including the following procedures: giving the panel 10 indication image information, the indication image information displays an indication image IP on the display screen of the panel 10 so that a finger exists; there is a finger along the indication image IP displayed on the display screen of the panel 10; the first transparent antenna 2 ₁ to the Nth transparent antenna 2 Among _N , there are multiple activated antennas around the position on the display screen of the panel 10 indicated by the indication image IP displayed on the display screen of the panel 10, which receive the reflected wave reflected by the finger, and receive signals from the multiple activated antennas. The position information (x, y) on the display screen of the panel 10 obtained by the indication image IP is bound to the received information S (t, g, h) corresponding to the transparent antenna that received the reflected wave used in the received received signal, and the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 is generated.

As described above, the operation panel device of the second embodiment has the _same function as the operation panel device of the first embodiment. Multiple antennas among _N receive reception signals from reflected waves reflected by the finger (here, the aforementioned multiple antennas exist around the position on the display screen of the panel 10 indicated by the indication image IP displayed on the display screen of the panel 10 and are activated), and the position information (x, y) on the display screen of the panel 10 obtained by the indication image IP is bound to the reception information S (t, g, h) corresponding to the transparent antenna that received the reflected wave, so as to generate teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422. Therefore, the transmission wave is radiated by the transparent antenna around the finger, and the reception intensity of the reflected wave of the transmission wave reflected by the finger is higher at the transparent antenna around the finger, so that the teacher information S (t, g, h | x, y) can be efficiently generated, and the memory capacity of the teacher information storage unit 422 can be efficiently used.

[Third embodiment] The operation panel device of the third embodiment is described using FIG. 10. The operation panel device of the third embodiment differs from the operation panel device of the first embodiment in that a receiving sensitivity determination unit 43 is provided, and the other points are the same. The receiving sensitivity determination unit 43 detects the receiving sensitivity of the reflected wave from the finger based on the transparent antenna _2k , and reminds to obtain the teacher information S (t, g, h | x, y) again when the receiving sensitivity becomes below the set value. By detecting the degradation of the receiving sensitivity, it can be known that the degradation of the receiving sensitivity of the reflected wave from the finger based on the transparent antenna _2k is caused by dirt attached to the display screen of the panel 10 or cracks and ruptures on the display screen of the panel 10. In addition, in FIG. 10, the same symbols as those added in FIGS. 1 to 8 represent the same or corresponding parts.

The following description will focus on the differences between the operation panel device of the first embodiment. In the operation panel device of the third embodiment, the position estimation action and the calibration execution action, that is, the action of generating calibration result data additionally bound to the position information (x, y) on the display screen of the panel 10 as the teacher information S (t, g, h | x, y), are the same as the position estimation action according to the flowchart shown in FIG. 7 and the calibration execution action according to the flowchart shown in FIG. 8 in the operation panel device of the first embodiment, so the description thereof is omitted.

Therefore, the following mainly describes the point that the reception sensitivity determination unit 43 detects the degradation of the reception sensitivity of the reflected wave from the finger based on the transparent antenna _2k , and prompts to obtain the teacher information S (t, g, h | x, y) again when the reception sensitivity becomes less than the set value. The reception sensitivity determination unit 43 obtains the reception information S (t, g, h) obtained by the reception information acquisition unit 41, and in this example, obtains (N×N) reception information S (t, g, h) in each transmission cycle c.

The reception sensitivity determination unit 43 detects the strength of the received signal obtained from the transparent antenna _2k included in the obtained reception information S (t, g, h), and based on the detected strength of the received signal, displays a picture of reception sensitivity degradation, which reminds the user to obtain the teacher information S (t, g, h | x, y) again when the strength of the received signal becomes less than a set value, and provides the panel 10 via the teacher information generation unit 423 and the control unit 50, and displays the picture of reminder to obtain the teacher information again on the display screen of the panel 10. In addition, the reception sensitivity determination unit 43 may provide the panel 10 with the reception sensitivity degradation information via the control unit 50 without passing through the teacher information generation unit 423.

For example, the comparison between the strength of the received signal and the set value in the receiving sensitivity determination unit 43 is performed as follows. The maximum value of the strength of the received signal contained in each of the (N×N) received information S (t, g, h) in each transmission cycle c is measured, and the maximum value of the measured strength of the received signal is compared with the set value.

When a transmission wave is radiated from the transparent antenna _2k located near the position where a finger is present on the display screen of the panel 10, and the transparent antenna _2k located near the position where the finger is present on the display screen of the panel 10 receives a reflected wave from the finger, the intensity of the received signal from the transparent antenna _2k located near the position where the finger is present that has received the reflected wave generally becomes the maximum value among the intensities of the received signals obtained by the (N×N) received information S(t, g, h).

Therefore, if the maximum value of the intensity of the received signal obtained based on the (N×N) received information S (t, g, h) becomes less than the set value, it can be estimated that the display screen of the panel 10 where the finger exists is dirty or cracked, and as a result, it can be judged that the receiving sensitivity of the transparent antenna _2k located near the position where the finger exists on the display screen of the panel 10 is deteriorated.

In addition, the transparent antenna _2k located near the finger position and serving as a transmitting antenna is not necessarily identical to the transparent antenna _2k located near the finger position and serving as a receiving antenna. In addition, the received information S(t, g, h) that has the maximum value among the received signal strengths obtained from the (N×N) received information S(t, g, h) is not necessarily one.

In the above example, although the receiving sensitivity is performed by the strength of the received signal, it can also be performed by detecting the spectrum shape of the received signal. When the change of the spectrum shape exceeds the change of the set value, in other words, it becomes a spectrum shape below the set spectrum shape, the receiving sensitivity degradation information that reminds to obtain the teacher information S (t, g, h | x, y) again is generated. The receiving sensitivity determination unit 43 can also be assembled in the computer together with the receiving information acquisition unit 41 and the machine learning unit 42. In addition, as the receiving sensitivity determination unit 43, it can also include a sensor for detecting the strength of the received signal from the transparent antenna _2k , which is used to compare the strength of the received signal obtained by the sensor with the set value to obtain the receiving sensitivity degradation information.

When an image reminding to obtain teacher information again is displayed on the display screen of the panel 10, the user executes the acquisition of teacher information on the operation panel device. The calibration execution action of the operation panel device, that is, the action of generating calibration result data bound to the position information (x, y) on the display screen of the panel 10 as teacher information S (t, g, h | x, y), is the same as the action described in the first embodiment, and is executed according to the flowchart shown in Figure 8.

However, in each of the first setting position _P1 to the D-th setting position _PD , the teacher information S (t, g, h | x, y) generated by the teacher information generating unit 423, when stored from the teacher information generating unit 423 to the teacher information storing unit 422, overwrites the teacher information S (t, g, h | x, y) in each of the first setting position _P1 to the _D -th setting position PD stored in the teacher information storing unit 422.

In this way, when the receiving sensitivity of the reflected wave from the finger obtained based on the transparent antenna _2k deteriorates, the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 can be rewritten, so the finger position can be estimated by the optimal teacher information S (t, g, h | x, y) corresponding to the state of the display screen of the panel 10.

As described above, the operation panel device of the third embodiment has the same effect as the operation panel device of the first embodiment. In addition, the reception sensitivity determination unit 43 of the operation panel device of the third embodiment gives the panel 10 reception sensitivity degradation information when the reception sensitivity of the reception signal from the first transparent antenna ₂₁ to the Nth transparent antenna _2N is below the set value, and displays an image of prompting to obtain the teacher information S (t, g, h | x, y) on the display screen of the panel 10, so that the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 can be rewritten corresponding to the state of the display screen of the panel 10, so that the finger position can be estimated by the best teacher information S (t, g, h | x, y) corresponding to the state of the display screen of the panel 10.

[Fourth embodiment] The operation panel device of the fourth embodiment is described using FIG. 11. The operation panel device of the fourth embodiment is different from the operation panel device of the first embodiment in that a time elapsed determination unit 44 is provided, and the other points are the same. In addition, in FIG. 11, the same symbols as those attached to FIGS. 1 to 8 represent the same or equivalent parts.

Due to the influence of the operating environment of the operation panel device, over a long period of time, the received information S (t, g, h) obtained by the received signal (from the first transparent antenna ₂₁ to the Nth transparent antenna _2N that received the reflected wave from the finger) may be affected, and the accuracy of the machine learning performed by the position estimation unit 421 using the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 may be reduced. In view of this point, after the time determination unit 44, when the teacher information generation unit 423 generates all the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 and the elapsed time from the moment of storage in the teacher information storage unit 422 becomes longer than the set time, a reminder to obtain the teacher information S (t, g, h | x, y) is displayed on the display screen of the panel 10.

The following description will focus on the differences between the operation panel device of the first embodiment. In the operation panel device of the fourth embodiment, the position estimation action and the calibration execution action, that is, the action of generating calibration result data additionally bound to the position information (x, y) on the display screen of the panel 10 as the teacher information S (t, g, h | x, y), are the same as the position estimation action according to the flowchart shown in FIG. 7 and the calibration execution action according to the flowchart shown in FIG. 8 of the operation panel device of the first embodiment, so the description thereof is omitted.

The following mainly describes the part that after the time determination unit 44 completes the action of generating the teacher information S (t, g, h | x, y) in the teacher information generating unit 423, a reminder is given to obtain the teacher information S (t, g, h | x, y) again when the elapsed time becomes longer than the set time. The time determination unit 44 receives the time (hereinafter also referred to as the calibration end time) when the teacher information generating unit 423 completes the action of generating the teacher information S (t, g, h | x, y) from the teacher information generating unit 423, and stores the calibration end time.

The elapsed time determination unit 44 periodically reads the current time, compares the read current time with the stored calibration end time, and when the elapsed time from the calibration end time to the current time becomes longer than the set time, the elapsed time image information displayed as an image reminding to obtain the teacher information S (t, g, h | x, y) again is given to the panel 10 via the teacher information generation unit 423 and the control unit 50, and an image indicating the necessity of obtaining the teacher information again is displayed on the display screen of the panel 10. In addition, the elapsed time determination unit 44 may also give the elapsed time image information to the panel 10 via the control unit 50 without passing through the teacher information generation unit 423.

In the above example, although the elapsed time is obtained by using the calibration end time and the current time, the elapsed time determination unit 44 may also include a timer, and the timer is set by the calibration end time. When the elapsed time counted by the timer becomes the set time, the elapsed time image information is output. The elapsed time determination unit 44 may also be assembled in a computer together with the received information acquisition unit 41 and the machine learning unit 42.

When an image reminding to obtain teacher information again is displayed on the display screen of the panel 10, the user executes the acquisition of teacher information on the operation panel device. The calibration execution action of the operation panel device, that is, the action of generating calibration result data bound to the position information (x, y) on the display screen of the panel 10 as teacher information S (t, g, h | x, y), is the same as the action described in the first embodiment, and is executed according to the flowchart shown in Figure 8.

However, in each of the first setting position _P1 to the D-th setting position _PD , the teacher information S (t, g, h | x, y) generated by the teacher information generating unit 423, when stored from the teacher information generating unit 423 to the teacher information storing unit 422, overwrites the teacher information S (t, g, h | x, y) in each of the first setting position _P1 to the _D -th setting position PD stored in the teacher information storing unit 422.

In addition, when the action of the teacher information generating unit 423 to obtain the teacher information S (t, g, h | x, y) again is completed, the calibration end time is received by the teacher information generating unit 423 through the time determination unit 44, and the calibration end time is overwritten. If the time determination unit 44 has a timer, the time of the timer is reset.

As described above, the operation panel device of the fourth embodiment has the same function as the operation panel device of the first embodiment. Moreover, in the operation panel device of the fourth embodiment, the teacher information generating unit 423 generates the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422. When the time elapsed from the time of storage in the teacher information storage unit 422 is longer than the set time, the panel 10 is provided with the image information of the elapsed time, and the image of the reminder to obtain the teacher information S (t, g, h | x, y) is displayed on the display screen of the panel 10. Therefore, the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 can be rewritten every time longer than the set time, and the finger position can be estimated by the optimal teacher information S (t, g, h | x, y) considering the influence of the use environment of the operation panel device.

[Fifth embodiment] The operation panel device of the fifth embodiment is described using FIG. 12. The operation panel device of the fifth embodiment is different from the operation panel device of the first embodiment in that a device temperature determination unit 45 and a device temperature measurement unit 46 are provided, and the other points are the same. In addition, in FIG. 12, the same symbols as those attached to FIGS. 1 to 8 represent the same or equivalent parts.

In the operation panel device, if the temperature of the device changes to a value greater than a set value, the device temperature drift of the first transparent antenna ₂₁ to the Nth transparent antenna _2N is greater than a specified value, _and the first transparent antenna 21 to the Nth transparent antenna _2N receive reflected waves from the finger. The received information S (t, g, h) obtained from the received signals from the first transparent antenna ₂₁ to the Nth transparent antenna _2N is affected, so the accuracy of the machine learning performed by the position estimation unit 421 using the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 is reduced.

In view of this point, when the difference between the temperature of the device (the temperature of the panel 10 in this example) and the temperature of the device (the temperature of the panel 10 in this example) when the teacher information generating unit 423 generates and stores all the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 is greater than the set value, the device temperature determining unit 45 prompts to obtain the teacher information S (t, g, h | x, y) on the display screen of the panel 10.

The following description will focus on the differences between the operation panel device of the first embodiment. In the operation panel device of the fifth embodiment, the position estimation action and the calibration execution action, that is, the action of generating calibration result data additionally bound to the position information (x, y) on the display screen of the panel 10 as the teacher information S (t, g, h | x, y), are the same as the position estimation action according to the flowchart shown in FIG. 7 and the calibration execution action according to the flowchart shown in FIG. 8 of the operation panel device of the first embodiment, so the description thereof is omitted.

Therefore, the explanation will be mainly based on the point that the device temperature determination unit 45 compares the difference between the temperature of the panel 10 when the teacher information generating unit 423 generates and stores the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422 and the temperature of the panel at the set time when the position estimation action is performed, and reminds to obtain the teacher information S (t, g, h | x, y) again when the difference between the two temperatures is greater than the difference setting value.

The device temperature measuring unit 46 detects the temperature of the device (the temperature of the panel 10 in this example). The teacher information generating unit 423 generates the teacher information S (t, g, h | x, y) stored in the teacher information storage unit 422, and when stored in the teacher information storage unit 422, that is, at the end of the calibration execution action and after a set time has passed since the end of the calibration execution action, the temperature of the panel 10 detected by the device temperature measuring unit 46 is periodically obtained and given to the device temperature determining unit 45.

The device temperature determination unit 45 stores the temperature of the panel 10 when the calibration execution action from the teacher information generation unit 423 is completed as a set value. The device temperature determination unit 45 uses the temperature of the panel 10 periodically obtained by the teacher information generation unit 423 as a comparison value, and compares the stored set value with the comparison value. When the difference between the set value and the comparison value is greater than the difference set value, the device temperature determination unit 45 provides the panel 10 with temperature image information displayed as an image to remind the teacher information S (t, g, h | x, y) to be obtained again through the teacher information generation unit 423 and the control unit 50, and displays an image indicating the necessity of obtaining the teacher information again on the display screen of the panel 10.

In addition, although the device temperature determination unit 45 obtains the temperature of the panel 10 detected by the device temperature measurement unit 46 through the teacher information generation unit 423, it can also be directly obtained by the device temperature measurement unit 46. In addition, the device temperature determination unit 45 can also provide the temperature image information to the panel 10 through the control unit 50 without passing through the teacher information generation unit 423.

When an image reminding to obtain teacher information again is displayed on the display screen of the panel 10, the user executes the acquisition of teacher information on the operation panel device. The calibration execution action of the operation panel device, that is, the action of generating calibration result data bound to the position information (x, y) on the display screen of the panel 10 as teacher information S (t, g, h | x, y), is the same as the action described in the first embodiment, and is executed according to the flowchart shown in Figure 8.

However, in each of the first setting position _P1 to the D-th setting position _PD , the teacher information S (t, g, h | x, y) generated by the teacher information generating unit 423, when stored from the teacher information generating unit 423 to the teacher information storage unit 422, overwrites the teacher information S (t, g, h | x, y) stored in each of the first setting position _P1 to the D-th setting position _PD in the teacher information storage unit 422.

In addition, when the action of the teacher information generating unit 423 obtaining the teacher information S (t, g, h | x, y) again is completed, the device temperature determining unit 45 will overwrite the stored temperature setting value of the panel 10 with the temperature of the panel 10 obtained by the device temperature measuring unit 46 when the teacher information generating unit 423 obtains the teacher information S (t, g, h | x, y) again, and store it as a new setting value.

As described above, the operation panel device of the fifth embodiment has the same effect as the operation panel device of the first embodiment. In addition, in the operation panel device of the fifth embodiment, the device temperature determination unit 45 compares the temperature of the panel 10 with the temperature setting value of the panel 10. When the difference between the two temperatures is greater than the difference setting value, the panel 10 is given time image information and an image prompting to obtain the teacher information S (t, g, h | x, y) is displayed on the display screen of the panel 10. Therefore, the device temperature drift for the first transparent antenna ₂₁ to the Nth transparent antenna _2N can be suppressed within the specified value, and the finger position can be estimated by the teacher information S (t, g, h | x, y) without being easily affected by the temperature of the operating environment of the operation panel device.

In addition, any of the receiving sensitivity determination unit 43 shown in the third embodiment, the elapsed time determination unit 44 shown in the fourth embodiment, or the device temperature determination unit 45 shown in the fifth embodiment can be applied to the operation panel device of the second embodiment. In addition, the embodiments can be freely combined, or any constituent elements of the embodiments can be deformed, or any constituent elements in the embodiments can be omitted.

<Industrial Applicability> The disclosed operation panel device is a non-contact operation panel device, which is suitable for use as a user interface for various electronic devices with touch display functions (such as mobile terminals and portable devices). In addition, it is suitable as an operation panel device for use in an environment such as a factory where dirt is easily attached to the display screen of the panel, and is used by users with dirty fingers.

10: display panel 20: transparent antenna array 2 ₁₁ ~2 _mn ,2 ₁ ~2 _N : first transparent antenna to Nth transparent antenna 30: transmission and reception circuit section 31: input and output switching section 31 ₁ ~31 _N : first switching section to Nth switching section 32: signal transmission section 321: signal generator 322: output destination selection section 33: reception information generation section 33 ₁ ~33 _N : First signal receiving unit to Nth signal receiving unit 40: Object position estimation unit 41: Reception information acquisition unit 42: Machine learning unit 421: Position estimation unit 422: Teacher information storage unit 423: Teacher information generation unit 43: Reception sensitivity determination unit 44: Elapsed time determination unit 45: Device temperature determination unit 46: Device temperature measurement unit 50: Control unit Tx1~TxN: Rising frequency modulation signal IP: Indication image IPa: Light drawing line IPb: Dark drawing line ST11~ST16, ST21~ST27: Step

FIG. 1 is a block diagram illustrating an operation panel device of the first embodiment. FIG. 2 is a model diagram illustrating a surface of a panel on which a transparent antenna array is embedded in the operation panel device of the first embodiment. FIG. 3 is an explanatory diagram illustrating up-conversion frequency modulation signals Tx(1) to Tx(N) generated by a signal generator of the operation panel device of the first embodiment. FIG. 4 is a schematic front view illustrating the relationship between an indication image displayed on a display screen of the panel and a finger in the operation panel device of the first embodiment. FIG. 5 is a hardware configuration diagram implemented by dedicated hardware, illustrating a first example of an object position estimation unit 40 of the operation panel device of the first embodiment. FIG. 6 is a hardware configuration diagram of a computer implemented by software or firmware, etc., illustrating a second example of an object position estimation unit 40 of the operation panel device of the first embodiment. FIG. 7 is a flow chart illustrating a position estimation action processing procedure for performing finger position estimation processing in the operation panel device of the first embodiment. FIG. 8 is a flow chart illustrating a calibration execution action processing procedure for generating teacher information in the operation panel device of the first embodiment. FIG. 9 is a schematic front view illustrating the relationship between the indication image displayed on the display screen of the panel and the finger in the operation panel device of the second embodiment. FIG. 10 is a block diagram illustrating the operation panel device of the third embodiment. FIG. 11 is a block diagram illustrating the operation panel device of the fourth embodiment. FIG. 12 is a block diagram illustrating the operation panel device of the fifth embodiment.

10: Display panel

20: Transparent antenna array

2 ₁ ~2 _N : 1st transparent antenna to Nth transparent antenna

30: Transmitting and receiving circuit section

31: Input and output switching unit

31 ₁ ~31 _N : first switching unit to Nth switching unit

32: Signal transmission department

321:Signal generator

322: Output destination selection unit

33: Receiving information generation department

33 ₁ ~33 _N : first signal receiving unit to Nth signal receiving unit

40: Object position estimation unit

41: Receiving information acquisition department

42: Machine Learning Department

421: Location Estimation Department

422: Teacher information storage department

423: Teacher Information Generation Department

50: Control Department

Claims (14)

An operation panel device comprises: A display panel having a display screen; A plurality of transparent antennas arranged two-dimensionally on the display screen of the display panel; A reception information generating unit generating reception information additionally bound to the transparent antenna receiving the reflected wave and corresponding to the plurality of transparent antennas based on the reception signal from the plurality of transparent antennas, wherein the plurality of transparent antennas receive the reflected wave reflected from the object existing on the display screen of the display panel; A storage unit storing teacher information additionally bound to the position information on the display screen of the display panel and corresponding to the plurality of transparent antennas; and The position estimation unit uses the teacher information stored in the storage unit to perform machine learning on the received information generated by the received information generation unit to estimate the position of the object on the display screen of the display panel. The operation panel device as claimed in claim 1 further comprises: A teacher information generating unit, which provides the display panel with indication image information, wherein the indication image information displays an indication image that causes the object to exist on the display screen of the display panel, the object exists along the indication image displayed on the display screen of the display panel, the plurality of transparent antennas receive the reflected waves reflected by the object, based on the received signals from the plurality of transparent antennas, the position information on the display screen of the display panel indicated by the indication image is attached to the received information corresponding to the transparent antenna that receives the reflected waves, and the teacher information stored in the storage unit is generated. The operation panel device as claimed in claim 1 further comprises: a teacher information generating unit, providing the aforementioned display panel with indication image information, wherein the aforementioned indication image information displays an indication image on the display screen of the aforementioned display panel so that the aforementioned object exists, the indication image displayed on the display screen of the aforementioned display panel indicates a position on the display screen of the aforementioned display panel, and among the aforementioned plurality of transparent antennas, activating a plurality of transparent antennas existing around the aforementioned position indicated by the aforementioned indication image, the aforementioned object exists along the aforementioned indication image displayed on the display screen of the aforementioned display panel, the aforementioned plurality of activated transparent antennas receive reflected waves reflected by the aforementioned object, Based on the reception signals from the aforementioned activated plurality of transparent antennas, the position information on the display screen of the aforementioned display panel indicated by the aforementioned indication image is attached to the reception information corresponding to the transparent antenna that received the aforementioned reflected wave, thereby generating the teacher information stored in the aforementioned storage unit. The operation panel device as claimed in claim 1 further comprises: A teacher information generating unit, which provides the display panel with indication image information, wherein the indication image information displays an indication image of a trajectory of the object moving on the display screen of the display panel, the object moves along the trajectory of the indication image displayed on the display screen of the display panel, the plurality of transparent antennas receive reflected waves reflected by the object, based on the received signals from the plurality of transparent antennas, the position information on the display screen of the display panel obtained by binding the indication image is attached to the received information corresponding to the transparent antenna that receives the reflected waves, and the teacher information stored in the storage unit is generated. An operation panel device as claimed in any one of claims 1 to 4, wherein the machine learning performed by the aforementioned position estimation unit uses any machine learning method of deep learning or Gaussian process regression. The operating panel device of any one of claim items 2 to 4 further includes: a receiving sensitivity determination unit, which provides the display panel with receiving sensitivity degradation information when the receiving sensitivity of the received signal from the aforementioned multiple transparent antennas is below the set value, and displays an image on the aforementioned display screen to remind the user to obtain the aforementioned teacher information. The operation panel device of any one of request items 2 to 4 further includes: an elapsed time determination unit, the aforementioned teacher information generation unit generates teacher information stored in the aforementioned storage unit, and when the elapsed time from the moment stored in the aforementioned storage unit is longer than the set time, the elapsed time image information is given to the aforementioned display panel and an image is displayed on the aforementioned display screen to remind the user to obtain the aforementioned teacher information. The operation panel device of any one of claim items 2 to 4 further includes: a device temperature determination unit, when the difference between the temperature of the aforementioned operation panel device and the temperature of the aforementioned operation panel device when the aforementioned teacher information generating unit generates the aforementioned teacher information stored in the aforementioned storage unit is greater than a set value, then the aforementioned display panel temperature image information is given and an image reminding to obtain the aforementioned teacher information is displayed on the aforementioned display screen. A method for estimating the position of an object, for estimating the position of an object on a display screen of an operation panel, comprises: a step of generating received information, wherein a plurality of transparent antennas arranged two-dimensionally on the display screen of the display panel receive reflected waves reflected from the object existing on the display screen before the display panel, a received information generating unit receives received signals from the plurality of transparent antennas, and based on the received received signals, generates the received information additionally bound to the transparent antennas that received the reflected waves and corresponding to the plurality of transparent antennas; and a step of estimating the position, wherein the position estimating unit uses teacher information stored in a storage unit to perform machine learning on the received information generated by the received information generating unit to estimate the position of the object on the display screen before the display panel. The object position estimation method of claim 9 further includes: A step of generating teacher information, wherein the teacher information generating unit gives the display panel indication image information, the indication image information displays an indication image that makes the object exist on the display screen of the display panel, the object exists along the indication image displayed on the display screen of the display panel, the plurality of transparent antennas receive reflected waves reflected by the object, the teacher information generating unit receives reception signals from the plurality of transparent antennas, and based on the received reception signals, the position information on the display screen of the display panel obtained by binding the indication image is attached to the reception information corresponding to the transparent antenna that received the reflected waves, thereby generating the teacher information stored in the storage unit. A program for estimating the position of an object, causing a computer to execute: A program for acquiring received information, wherein a plurality of transparent antennas arranged two-dimensionally on a display screen of a display panel receive reflected waves reflected from an object existing on the display screen of the display panel, and based on received signals from the plurality of transparent antennas, the received information corresponding to the plurality of transparent antennas and additionally bound to the transparent antennas that received the reflected waves is acquired; and A program for estimating a position, wherein machine learning is performed on the acquired received information using teacher information to estimate the position of the object on the display screen of the display panel. The program for estimating the position of an object as claimed in claim 11 further comprises the following steps: A program for generating teacher information, wherein the display panel is provided with indication image information, the indication image information displays an indication image that makes the object exist on the display screen of the display panel, the object exists along the indication image displayed on the display screen of the display panel, the plurality of transparent antennas receive reflected waves reflected by the object, and based on the received signals from the plurality of transparent antennas, the position information on the display screen of the display panel obtained by binding the indication image is attached to the received information corresponding to the transparent antenna that received the reflected waves, thereby generating the teacher information. A recording medium stores a program for estimating the position of an object on a display screen of an operation panel device, the program causing a computer to execute: A program for acquiring received information, wherein a plurality of transparent antennas are two-dimensionally arranged on a display screen of a display panel, the plurality of transparent antennas receive reflected waves reflected from the object existing on the display screen of the display panel, and based on received signals from the plurality of transparent antennas, the received information additionally bound to the transparent antennas that received the reflected waves and corresponding to the plurality of transparent antennas is acquired; A program for estimating a position, wherein machine learning is performed on the acquired received information using teacher information to estimate the position of the object on the display screen of the display panel. The recording medium of claim 13 further includes the following steps: A procedure for generating teacher information, wherein the display panel is provided with indication image information, the indication image information displays an indication image that causes the object to exist on the display screen of the display panel, the object exists along the indication image displayed on the display screen of the display panel, the plurality of transparent antennas receive reflected waves reflected by the object, and based on the received signals from the plurality of transparent antennas, the position information on the display screen of the display panel obtained by binding the indication image is attached to the received information corresponding to the transparent antenna that received the reflected waves, thereby generating the teacher information.

Images