JP4708278B2 - Touch panel system and operation method thereof - Google Patents

Description

  The present invention relates to a touch panel system and an operation method thereof, and more particularly, to a large screen rear projection system capable of interacting with a computer by touching a specific portion of a projection image with a finger and an operation method thereof.

  As activities such as business, exhibitions, demonstrations and audiovisual education become more frequent, large screen projection technology has already become an indispensable multimedia display means for people, and its facilities are already in scientific research, educational process, military training, It has come to be widely applied in various fields such as industry application.

  New technologies appear one after another, providing a potential condition for the continuous development of large-screen projection technology, and projectors can be used from the initial low brightness (less than 800 lumens) to the current high brightness (more than 5000 lumens). And high resolution (1024 × 768 or higher), the material of the large screen is changed from the initial white wall medium (gain 0.8) to the white plastic screen with gain 0.9, and then the glass screen with gain of about 2 It has been developed to a metal screen that has recently reached a gain of 8. However, the development of the projector and the improvement of the material of the large screen are all for enhancing the image effect of the projection screen system.

  However, most of the screen projection systems in the existing technology are unidirectional, and interactive information inquiry functions such as a touch panel are not possible. Yes. In particular, it is impossible for several people in different locations to work on the same content through an online computer during a long distance video conference. Conventional touch panel screen technology includes four types: resistance type, capacitance type, ultrasonic type and infrared type, but it is usually impossible to realize a large size, and there is a high demand for the use environment, and it is still a large screen. There is no touch panel that fits the rear projection screen.

  An object of the present invention is to use a visual sensor to sense and identify the position of a finger or pen on a rear projection large screen screen, convert it to computer command information, and bring the touch screen function to the projection screen That is.

In order to achieve the above object, the touch panel system of the present invention includes a transparent column, a projection screen, a projector, and a computer. The system further includes a visual sensor that is connected to a digital camera, a camera interface, a digital signal processor and a computer interface, a digital camera and a camera interface, and the camera interface is connected to the digital signal processor, A signal output from the digital signal processor is connected to the computer by a computer interface. The projection screen is installed on a transparent column, the visual sensor is installed between the projection screen and the projector, the visual sensor and the projection screen are kept at the same height, and both the visual sensor and the projector are connected to the computer.
The visual sensor can be installed at the same height in front of the projection screen, the number of digital cameras is one or more, and the material of the projection screen is polystyrene.

The operation method of the touch panel system of the present invention includes the following steps.
(1) Start-up correction (2) Pick up the feature points of the projection screen and lower the dimension.
(3) The expert system sifts through rudimentary.
(4) Determine whether the information meets the definition condition.
(5) If there is relevant information, advance to the neural network calculation method.
(6) It is determined whether or not there is a finger position.
(7) If there is, extract the finger position.
(8) The extracted finger position is compared with the information transmitted from the visual sensor.
(9) The computer performs a comparison process of information and performs control while driving the mouse.

  A five-point localization method is used for start-up correction (calibration processing for matching computer screen coordinates and projection screen image coordinates at start-up). Specifically, when the computer cursor moves to the upper left corner-upper right corner-screen center point-lower left corner-lower right corner and the computer cursor stops at each position, the user moves the cursor position on the screen with the laser pen. Move the cursor to the next position. Proceed to the next position and repeat the above procedure until all five points are reached.

  The expert system screens the image information on the projection screen and determines the finger position by judging the finger information on the image (fingertip images of people who may use the screen or an unspecified number of people). Identify. The expert system includes a total database (knowledge database), control strategy and generative rules (inference engine). The knowledge database contains certain fingers that are generated under image analysis that are recognized during practice. Experience information and data, and some data information that can represent the finger information in the image during the image processing process are included, and the control strategy is the finger information identification of how to reduce errors in various environments as much as possible It is a calculation method, and the generation rule is responsible for the coordination and harmonization of the total database and control strategy.

  As a neural network calculation method, Mean Taxonomy is adopted as a hidden layer learning method, and a gradient descent method is applied as an output layer learning method.

  Embodiments of the present invention will be described below with reference to the drawings. However, even if there are specific descriptions in the dimensions, materials, shapes, relative arrangements, and the like of the component parts described in the present embodiment, the present invention is not intended to be limited thereto.

  FIG. 1 is a conceptual diagram of the touch panel system of the present embodiment. A transparent column 1, a projection screen 2, a visual sensor 3, a projector 4 and a computer 5 are included. The material of the transparent column 1 can be glass or the like, and the material of the projection screen 2 is polystyrene, and it can be installed on the transparent column 1 to reveal the terminal and the touch screen interface. The visual sensor 3 is installed between the projection screen 2 and the projector 4, and the height thereof may coincide with the projection screen 2 or directly below the projection screen 2. The visual sensor realizes sensing for the area of the projection screen 2 by specular reflection of the plane mirror several times (not shown in the drawing). Both the projector 4 and the visual sensor 5 are connected to the computer 5 through a standard RS232C interface or USB interface to realize a communication function between them.

  In FIG. 2, the visual sensor 3 includes a digital video camera CAMERA 31, a camera interface CAMERA • I / F 32, a digital signal processor DSP 33, and a computer interface COMP • I / F 34. One or a plurality of cameras 31 can be installed, and the role of the camera 31 is to acquire object information such as a finger of the projection screen 2, and the camera interface 32 receives the information collected by the camera 31 and receives a digital signal. The digital signal processor 33 digitizes the collected signal, identifies object features on the screen, and determines whether it is user input information or noise. Regarding the user input information, the physical coordinates on the screen of the user's finger or other object and the control command are calculated, and then transmitted to the computer 5 through the computer interface 34, and an operation corresponding thereto is executed.

FIG. 3 is a flowchart of the operation method of the touch panel system of the present embodiment. This method of operation includes the following steps.
Step 101: Start correction Step 102: Pick up the feature points of the projection screen and lower the dimension.
Step 103: The expert system sifts through rudimentarily.
Step 104: It is determined whether or not the information satisfies a definition condition.
Step 105: If there is relevant information, the process proceeds to the neural network calculation method.
Step 106: It is determined whether or not there is a finger position.
Step 107: If there is, perform finger position extraction.
Step 108: Compare the extracted finger position with the information transmitted from the visual sensor.
Step 109: The computer performs a comparison process of information and performs control while driving the mouse.

  In step 101, initial localization and correction are performed by a five-point localization method. Therefore, when starting the correction program, the computer cursor moves from the upper left corner to the upper right corner, the screen center point, the lower left corner, and the lower right corner. Each time the computer cursor stops at each position, the user moves the laser pen to the cursor position on the screen for a few seconds and the cursor moves to the next position. Proceed to the next position and repeat the above procedure until all five points are reached. After such a correction, when the finger touches the screen, the cursor stops just below the finger position as the finger moves.

  In step 102, a feature point of the projection screen (a point different from the image projected by the projector 4) is picked up and the dimension is lowered.

  In step 103, the expert system identifies finger information on the image to identify the finger position. The expert system includes a total database, control strategy and generative rules. The data in the total database can be represented by (1) certain experience information and data held by the finger that is generated under image analysis recognized during practice, and (2) finger information in the image during the image processing process. Some data information is included. Control strategy is a calculation method of finger information identification to reduce mistakes as much as possible in various environments. The generation rule is “If this condition is met, an operation should be taken. ], And is responsible for coordination and harmonization of the total database and control strategy. The expert system says, "If the data of a point on the image corresponds to some feature data in the database, this point is recognized that the finger information may be promoted to the next stage of screening. . Points that do not apply are excluded from sieving. The final conclusion is obtained after multiple sieving. ] System.

  In step 104, it is determined whether the information meets the finger conditions defined by the expert system.

  Step 105: If there is relevant information, the process proceeds to the neural network calculation method. The neural network calculation method in step 105 adopts the basic calculation method during model identification. That is, the neural network is trained and learned using the finger information obtained during the experiment, and the network parameters (for example, the number of hidden layer nodes, weight, hidden layer node size, etc.) are determined by such learning. From this parameter, the network structure is determined. Learning is to estimate and adjust network parameters according to some already obtained input / output information. As learning progresses, these parameters change a small amount around a central value. These central values can be established as structural parameters of the network and can be divided into roughly two stages in the whole learning process. A hidden layer learning process and an output layer learning process. The hidden layer training course is a process of determining the number of hidden layer nodes and weights, and connected weights can be obtained through training of the output layer. Among these, the learning method of the hidden layer can employ Mean Taxonomy, and the learning method of the output layer employs the gradient descending method, Gradient Decent (the steepest descent method). A more accurate position of the finger is determined by identifying the data transmitted from the visual sensor 3 by a software method in which the neural network and the expert system are mutually coupled.

  In step 106, it is determined whether or not there is a finger position.

  If there is a finger position in step 107, finger position extraction is performed.

  In step 108, the extracted finger position is compared with the information transmitted from the visual sensor.

  In step 109, the computer performs a process of checking information and performs control while driving the mouse.

  Here, the operation of the touch panel of this embodiment will be summarized. When a finger or a finger-shaped object operates on the projection screen 2, the visual sensor 3 captures an image scene on the projection screen 2 under a complicated background, converts it to a digital signal, and features of the object on the screen Is identified, whether it is user input information (finger information) or interference information (noise). Regarding user input information, the user's finger or other object calculates physical coordinates and control commands on the screen, and then transmits the target position and control information to the computer 5 for processing. Control of the mouse position and control information is performed by driving application software corresponding to the computer 5. The image information stored in the computer 5 is projected on a large screen projection screen through a projector, so that interactive interaction between the computer and a person can be achieved.

  According to this embodiment, the super projection screen can be changed to a contact screen.

  The sensor can be installed within a range of 1 m to 4 m behind the projector, and the height of the sensor installation is about 4 m from the ground.

  In addition, the size of the projector screen is determined by a five-point localization method, and no unscheduled request is submitted to the sensor when changing the screen size.

  Also, the five-point localization method is simple and accurate.

  [Supplement] The Chinese application described above will be supplemented.

  First, the digital signal processor (DSP 33) will be supplemented with reference to FIG.

  The digital signal processor is responsible for processing digital signals transmitted from the camera interface, calculating the position of the fingertip on the screen, and transferring information to the computer through the computer interface.

  The information processor reads the information that the camera interface has processed the image signal, where it calculates the characteristics of the digital image of the finger, identifies the finger from the background image, calculates the coordinates of its position, and calculates The computer interface delivers the result, that is, the information that the finger appears on the screen and the position information to the computer. At the same time, the information processor completes the distortion-rectifying calculation method of the visual system. Among them, the power management module is responsible for supplying power to the information processor. The operating status of the information processor is constantly monitored, and the power management module protects when the load is heavy or the voltage is unstable.

  The Clock Generator becomes an image elementary signal depending on the input video signal. Alternatively, if one image element is acquired, a signal is output once. In this system, the Clock Generator generates two different frequencies, 27 MHz and 13.5 MHz.

  The SDRAM temporarily stores information obtained by processing the image signal by the video camera interface. When the information processor finishes calculating one frame of image data, the stored contents of the SDRAM are updated.

  The FLASH memory stores program code inside the information processor. The data in the FLASH memory is read into the information processor, and later, these program codes are executed from the start to perform operations on the data temporarily stored in the SDRAM.

  With reference to FIG. 6, it supplements about a computer interface (COMP * I / F34). The computer interface transfers the calculation result of the information processor to the computer 5 through a serial port. In addition, through the serial port, the user can set the operation mode of the system according to the actual operation environment of the system. Thus, the system can be adapted to different environments. The program driver adjusts the corresponding parameters according to different modes so that target recognition and position measurement are achieved within the expected accuracy range. Using an asynchronous UART electric circuit and Level Translating Digital Bus Switches, it is connected to an interface of an external storage device of information digital, and is responsible for maintaining communication protocol matching.

  With reference to FIG. 7, it supplements about a camera interface (CAMERA * I / F32). The camera interface is in charge of collecting and adjusting the transfer signal of the camera 1, and is a signal format recognition unit, a synchronization signal inspection / measurement, a video signal inspection / measurement / output unit, a clock generator, and an A / D (including analog / digital converter. Signal format recognition determines the input signal to detect and measure the video signal. The system uses the PAL format.

  The A / D converter converts the analog signal of the camera to ITU-R BT. It is also converted into a digital signal of 656 standard format, and a luminosity digital signal and a chromaticity digital signal are also included. Therefore, it can be processed directly into a digital signal for later signal processing.

  Synchronous data transfer electrical circuit: Since the detection and measurement of the video synchronization signal refers to the periodic output representing the signal, signal processing tells the position of the current processing signal of the electrical circuit within the overall project. Frame synchronization, row synchronization, and odd / even synchronization signals are included. Frame synchronization is to send a synchronization signal for each frame image, row synchronization is to send a signal for each row image, and odd / even synchronization is a signal to be transferred indicating an odd field signal or an even field signal. That is.

  The Clock Generator becomes an image elementary signal depending on the input video signal. Alternatively, if one image element is acquired, a signal is output once. In this system, the Clock Generator generates two different frequencies of 27 MHz and 13.5 MHz.

  Next, it supplements about the flowchart of FIG.

  After the information of each basic point on the screen is determined by the five-point localization method of Step 101, the whole screen area is divided into different child areas in order to change the physical coordinates to logic coordinates. The child region has a fixed coefficient for size conversion.

  It supplements about step 102 (The feature point of a projection screen is picked up and a dimension is reduced). In the touch screen system of this embodiment, a principal component analysis method (Principal Component Analysis, abbreviated as PCA) is used for digital image signals. PCA uses a square variation. In this system, due to the disturbance caused by people and voltage, the signal obtained by the video camera is multi-dimensional and the information of finger characteristics is relatively complex, if all information is classified If done, the analysis speed of the system will be slow. Therefore, PCA reserves data with a large amount of information (large dispersion) through linear transformation, removes noise, and lowers the data dimension.

In the sample {Xi} (i is 1 to N) in the digital image data space, (information of pixels of pixels in one small range area of one frame image which is one of the digital image information) gray) and chromaticity (information of chromaticity) to obtain its mean X ₀ (mean value) and covariance Σ = E (X−X ₀ ) × (X−X ₀ ). Σ = UΛU ^T , where U is an orthogonal matrix and Λ is a diagonal matrix, a characteristic matrix of Σ The main component Y = U ^T (X−X ₀ ) is , One new data set Yi) (where i is 1 to N) The value is zero, lambda is the covariance matrix (covariance matrix).

  In this way, the association between variables is removed, variables with smaller square deviations are removed, thereby reducing the dimension of the data. Useful information is extracted from the grayness and chromaticity of all pixels within this range, and the finger feature information is limited to grayness, shape and movement information, and prepared for subsequent processing of the expert system.

  With reference to FIG. 8, it supplements about the expert system described in step 103. FIG. An expert is a person who has expertise in a specific area. Alternatively, an expert is a person who has specialized skills that other people do not know or use. Experts can solve problems that ordinary people cannot solve or cannot solve with high efficiency. An expert system is a branch of artificial intelligence, and uses a great deal of expertise to solve problems that only experts can solve.

The user provides facts or other information to the expert system and receives expert knowledge or expert advice in return. The interior of the expert system contains two parts: the main database and the reasoning engine. The database contains the conclusion knowledge to be acquired using the reasoning engine. This conclusion is output to the user from the expert system.
The expert system recognizes the finger position by judging the finger information on the image. It also includes a total database, control strategy and generation formula rules. The total database data includes (1) certain experience information and data held by fingers generated under image analysis recognized during practice, ( 2) Some data information that can represent the finger information in the image is included in the image processing process, and the control strategy is a calculation method of finger information identification to reduce the error in various environments as much as possible. The generation rule is “If this condition is met, an operation should be taken. ] And is responsible for coordination and harmonization of the total database and control strategy. The expert system actually recognizes that if the data at a point on the image corresponds to some feature data in the database, this point may be transferred to the next sieving stage. Is done. Points that do not apply are screened out. The final conclusion is obtained after multiple sieving. ].

  In step 104, useful information is extracted from the grayness and chromaticity of all the pixels within this range, and the finger feature information is limited to grayness, shape and movement information, and the condition defined in advance as a finger. A rough judgment is made as to whether or not this is true.

  With reference to FIG. 9, it supplements about a neuron (nerve cell).

  In the human brain, each cell extends a number of processes and is connected to fellow neurons (target cells). Axons are connected to many fellow cells while branching many times along the way, and serve to send out information. Innumerable trees are dendrites, which receive information from other cell axons. Each nerve cell in the brain, which is said to be 100 billion or 140 billion, is connected to an average of approximately 10,000 fellow cells, and the connections between the neurons that are spread like the mesh. Is a neural network.

  All the information obtained through the five senses is transmitted as electrical signals to this neural network. However, the axon-dendritic junction (synapse) has a very small gap called a synaptic gap, and it cannot be jumped over with an electrical signal. Therefore, when it reaches the end of the axon, it is converted into a chemical substance (neurotransmitter) in an amount corresponding to the signal and jumps out into the synaptic gap, and when the dendrite receptor (receptor) receives it, it quickly turns into electricity. It returns to the signal and is transmitted to the next nerve cell. In other words, in the brain, there are always a vast number of neural networks, and various types of information circulate while changing the form of electrical signals-> chemical substances-> electrical signals->. And we humans think, memorize, remember, imagine, and act by the function of the neural network.

Artificial neural nets (neural nets) are based on models of the physiological structure of nerve cells. The simplest model of a neuron is an artificial neuron that: x _{1 to} x _N are input signals from other neurons. Let w _{1 to} w _{N be the} intensity of sudden touch, and Y be the output signal of the neuron. All the above variables and parameters are real numbers. A simple input-output correspondence is as follows.

  Among them, sgn [•] represents a symbol value. This is the McCulloch-Pitts model published in the 1940s. Based on a particular neuron model, many neurons can synthesize different types of artificial neural networks in different ways.

The neural network has the following characteristics.
1 A network consists of one simple non-linear processor unit, that is, a neuron.
2 All networks are organized and operated in a distributed-parallel manner. Each neuron and dendrite takes over the memory of all memory contents, and even if all the memory is retrieved, the three networks related to all neurons and dendrites have a large number of neurons. The network can be a complex system.
4 The network has learning ability. It is a program that has already been made, and it comes to acquiring this function through learning rather than realizing the necessary function.

  The neural network calculation method of this touch panel system adopts the basic calculation method during model identification. The neural network is trained and learned using the finger information obtained during the experiment. By such learning, the parameters of the neural network (eg, number of hidden layer nodes, weight, hidden layer node size, etc.) are determined, From this parameter, the structure of the neural network is determined. Learning is to estimate and adjust the parameters of a neural network based on some already obtained input / output information. As learning progresses, these parameters change a small amount around a central value. These central values can be established as structural parameters of the network and can be divided into roughly two stages in the whole learning process. A hidden layer learning process and an output layer learning process. The hidden layer training process is the process of determining the number of hidden layer nodes and the right value, and the connected right value can be obtained through the output layer training. Among these, the learning method for the hidden layer can employ the average value collection method, and the learning method for the output layer employs the steepest descent method.

  In step 107, the mouse is replaced with a finger to control a large screen, so after dividing the handprint area, the position of the fingertip is measured in the handprint area and converted to screen coordinates. To give instructions. When the inspection / measurement is performed on one image, since the bill area is already divided, the search of the fingertip is changed from the entire area to the local search, and the search time can be reduced. This system uses the fingertip positioning technique that combines the method of calculating the maximum curvature point and the directed search method in the divided bill area.

  In this system, in order to obtain image information of the bill area divided by the above method, the coordinates (xc, yc) and the direction angle θc of the bill area center must be calculated. Thereby, an instruction is given to search for the direction of subsequent pattern matching.

First, the center coordinates of the bill area are calculated through M0 (zero-order rectangle) and M10 (first-order rectangle). The calculation formula for M0 (zero-order rectangle) and M10 (first-order rectangle) is as follows:

Then, the two-dimensional direction of the image is calculated using M20 (second floor rectangle). The calculation formula is as follows.

There are already many different ways to measure corner points in an image. The calculation method of this system uses inspection / measurement based on corner points of pattern (template) matching. The principle is as follows. Using image (template) matching in image processing is one mature and practical technique. It is a very good way to use pattern matching in the inspection and measurement of edges, and it can also be used to measure corner points. The principle of measuring the corner points in the pattern matching method is as follows. Determine the relevance and similarity of all M × N regions of the assigned n × n pattern and image. Bretsch established the following pattern (template).

  When the above-described pattern plates (templates) are continuously rotated by 90 °, a total of eight patterns (templates) can be produced. By calculating the degree of association between the image and the eight sets of templates (templates), the corner point measurements (pixels) of all the images are obtained, and finally the maximum value pixel is selected as the corner point.

  Under ideal circumstances, this pattern board can inspect and measure all corner points. However, since corner points have a lot of features (points, internal angles, edge trapezoids), it is impossible to design a large number of templates (templates) that match all corner points. Therefore, this method cannot be used in complex images.

  In step 108, information can be contrasted and a technique for fusing a large amount of information can be employed to eliminate a single feature restriction of information. Since the image processing calculation method based on the high-pass filter is used, the Sobel division operator is used to divide the bill image, and the finger has a specific gray feature under the background of the large screen, The limit value division and Sobel division operators increase the degree of cooperation with each other, and can eliminate unclear phenomena. Furthermore, taking into consideration the characteristics of finger movement, a motion detection algorithm that combines the screen difference between the background and time is used. Then, the position of the fingertip is determined based on a technique that combines the method of calculating the maximum curvature point and the directed search method in the divided bill area. For tracking the fingertip position, this system incorporates a Kalman filter to create a movement transfer rectangle and a survey rectangle according to the rules of finger movement. In order to meet the requirement of real-time performance, the system uses a PCI-based image collection card, and the collected image is directly placed in the memory, thereby shortening the image collection time to the maximum. Furthermore, in order to increase the execution efficiency of the calculation, the present system can reduce the image processing time by inputting the ash degree image as the original image and directly processing it into the ash degree image.

  In step 109, after the information on the fingertip position is recognized, the physical position of the fingertip in the image is converted into the upper logical coordinate information on the large screen through the coordinate conversion and transferred to the computer command. It becomes.

  The neural network calculation method of this system adopts the basic calculation method during model identification. The neural network is trained and learned using finger information obtained during the experiment, and the parameters of the neural network (for example, the number of hidden layer nodes, right value, hidden layer node size, etc.) are determined by such learning. From this parameter, the structure of the neural network is determined. Learning is to estimate and adjust the parameters of the neural network based on some already obtained input / output information. As learning progresses, these parameters change a small amount around a central value. These central values can be established as structural parameters of the network and can be divided into roughly two stages in the whole learning process. A hidden layer learning process and an output layer learning process. The hidden layer training process is the process of determining the number of hidden layer nodes and the right value, and the connected right value can be obtained through the output layer training. Among these, the learning method for the hidden layer can employ the average value collection method, and the learning method for the output layer employs the steepest descent method.

  The goal of realizing this system is to control the large screen instead of the mouse with the finger, so after dividing the bill area, measure the position of the fingertip in the bill area, and then the screen coordinates And instruct the system. When the inspection / measurement is performed on one image, since the bill area has already been divided, the search of the fingertip is changed from the entire area to the local search, and the search time can be reduced. This system uses the fingertip positioning technique that combines the method of calculating the maximum curvature point and the directed search method in the divided bill area.

  Step 102 (picking feature points of the projection screen and reducing the dimension) will be described. In the touch screen system of this embodiment, a principal component analysis method (Principal Component Analysis, abbreviated as PCA) is used for digital image signals. PCA uses a square variation. In this system, due to the disturbance caused by people and voltage, the signal obtained by the video camera is multi-dimensional and the information of the finger features is relatively complex, so if all information is classified The analysis speed of the system will be slow. Therefore, PCA reserves data with a large amount of information (large dispersion) through linear transformation, removes noise, and lowers the data dimension.

  Here, it supplements further about an expert system.

  The knowledge database is a database that accumulates basic data of finger information calibrated by the five-point localization method and data that reaches recognition of the finger position on the projection screen. Furthermore, the result of judging the position of the finger as experience information is stored in this knowledge database and used for subsequent analysis.

  Control strategy is a technique to train and learn neurons in the hidden layer and output layer in order to extract finger information more easily.For example, in order to identify information other than finger information as noise, neurons in the hidden layer This is a technique for adding a connection weight to a neuron of a selected specific output layer to make the output value as close as possible to the actual finger information.

  The generation rule constitutes a part of a technique for removing noise by a conditional expression for screening whether or not a specific condition is satisfied in relation to a database to be referenced and a control strategy.

  The number of hidden layer contacts is the number of neurons belonging to the hidden layer and is also referred to as the number of hidden layer neurons. The contact area of the hidden layer is the fluctuation range in which the value of the parameter of the neuron belonging to the hidden layer varies within an allowable range.

As shown in FIG. 10, the hidden layer is a path (weights W ₀ , W ₁ , W _b ) that connects the input layer (I ₀ .I ₁ , Bias unit) and the output layer Y (I ₀ .or I ₁ ). It is.

With reference to FIGS. 11 and 12, general characteristics of the neural network including the hidden layer will be described. A certain element j receives the output yi of another element i as an input, and adds the weight wij as the input summation uj (3.1).
The output yj is expressed by the sum of the inputs and the monotonically increasing function f (3.2). The threshold is considered to be included as one of the weights. Here, the output function f may be a step function. Here, a sigmoid function is used as the output function f. Since the sigmoid function is a function that takes a continuous value from 0 to 1, it can be differentiated, and the problem can be solved analytically later. Next, generalize learning in neural circuits. Equation (3.3) The second term (negative term) in the square page on the right side is the desired output to be output by the output element j for a certain input vector c, Equation (3.3) in the square page on the right side When the first term (positive term) is the actual output of the output element j at that time, an error function E as shown in Equation (3.3) is considered as an evaluation of learning. This procedure for minimizing the error function is generally called a least mean square (LMS) method. Since the first term (positive term) in the square page on the right side of Equation (3.3) is determined by the strength of coupling between elements at that time, that is, the weight wij, the error function is also a function defined implicitly with respect to the weight. Become. Therefore, if a space where each weight value is used as an axis is considered, and if a value defined by this error function E is considered as a height, E gives an error curved surface as a hypersurface in the weight space. In order to reach the minimum value of the error surface from an arbitrary weight state, for example, each weight may be changed by an amount proportional to the partial differential term on the right side of Equation (3.4). This is equivalent to proceeding in the steepest inclination direction on the error curved surface, and such a learning rule is generally referred to as a steepest descent method.

  If the properties of the element are defined as in the formulas (3.1) and (3.2), the formula (3.4) can be expanded by the differential formula of the composite function (formula (3.5 ) Subscript c is omitted). If the expressions (3.1) and (3.2) are differentiated and substituted, the expressions (3.6) and (3.7) are obtained, so the expression (3.4) is eventually converted into the expression (3.8). It becomes. When the intermediate layer does not learn, the first term of the multiplication term on the right side of Equation (3.5) (the one obtained by partial differentiation of E by yj) can be easily obtained by differentiating Equation (3.3). 9). Therefore, a learning rule of Expression (3.10) is obtained from Expression (3.8). This is called a generalized delta rule. For example, when f is given by a sigmoid function, it is in the form of (3.12) from equation (3.11). In the method of Equation (3.11), the weight is changed only after all the input / output patterns are given. However, if η is sufficiently small, the weight is given every time each input / output is given like a perceptron. Is iteratively changed. That is, even with (3.13), the total change amount is almost equal to the steepest descent method.

Neural-related learning rules often use the steepest descent method. Both backpropagation and the Hopfield model use the steepest descent method. For problems aimed at minimizing (or maximizing) the energy function, it is a common practice to use the steepest descent method as a method of parameter update. First, suppose that a certain function E (w) is given as a standard to be minimized (or maximized). The purpose is to obtain w = [w ₁ , w ₂ ,..., W _N ] ^t that minimizes (or maximizes) this function E (w). First, an appropriate initial value is given to w, W is sequentially updated by a method such as (8.1) or (8.2) shown in FIG. Here, η is a positive value smaller than 1.

  With reference to FIG. 14, minimizing (or maximizing) the function E (w) by this method will be described. First, minimization will be described. FIG. 9 is a plot of E (w) cut by a plane parallel to the wi-E (w) plane passing through E (w (t)) at time t. At this time, it is considered how E (w) can be made smaller by updating one of the parameters wi. The partial differentiation of E (t) with respect to wi represents the inclination in the wi direction at this point E (w (t)). If the slope is negative, E (w) becomes smaller than the current value if wi is moved in the positive direction. Conversely, if the slope is positive, you can move wi in the negative direction. In other words, it is only necessary to move wi in the direction opposite to the sign of the partial differential with respect to wi of E (t). Therefore, the update amount of wi can be set as −η × (partial differential with respect to wi of E (t)). It ’s perfect. Individually, η is a positive value. t This is the basic principle. If E (w) maximization is considered, it is only necessary to reverse the update direction. Actually, there are many cases where E (w) has many undulations, and this is often not easy (referred to as a minimum solution or a local minimum). There is a contrivance to “add an inertia term” as a countermeasure against it.

[Example 1] Example 1 is more specific than the above-described embodiment in that an infrared light source is integrated with a digital video camera, and the projection screen is made of infrared rays. It is the point which is the acrylic which can permeate | transmit. The digital camera is a superordinate concept that combines a digital still camera and a digital video camera.

  In particular, by illuminating the projection screen with infrared light, the finger position can be accurately detected with little noise from an image projected with visible light. Further, when the infrared stray light is strong, the output of the infrared light source is increased to reduce detection noise. On the other hand, when the infrared stray light is weak, the output of the infrared light source is decreased to reduce power consumption.

Example 1 will be described below. When the visual sensor is working, it gets a continuous image. The image is processed by the video camera interface VC / I / F 32 and input to the digital signal processor DPS33. The DPS 33 recognizes a finger shape that touches the screen 1. The calculation method of this finger recognition will be explained again. The calculation method includes the following steps.
1. The vision sensor system acquires the image frame from the video camera and measures the area,
2. Process the image based on the measurement means and extract the features of the finger
3. 3. Filter potential fingers through expert system. 4. Determine if there is a finger position In some cases, the finger shape is verified through the neural network recognition algorithm, and the finger position is read in the image coordinates. Computer interface electrical circuit transmits detected information to computer

  The expert system is a system that includes rules for determining a finger shape based on an experience information database. The neural network is a part of an operation for collating a clear finger shape from a sample of prepared finger shape characteristic parameters.

  The serial port sends finger contact information and fingertip position to the computer. A driver operating in the Windows task loop transmits data from the serial port. When data is received, camera image coordinates are mapped to desktop logical coordinates. The coordinate parameters to be mapped are obtained from the five-point measuring method already described in the embodiment of the present invention.

  Example 2 In Example 2, as shown in FIG. 15, a transparent glass of an existing building show window was used as the transparent support column 1. The projection screen 2 was brought into close contact with the transparent glass. At this time, the transparent support column 1 and the projection screen 2 can be integrated by using a glass processed so that transmitted light is scattered so that at least one surface of the transparent glass can be used as a projection screen. The visual sensor 3, the projector 4, the computer 5, and the infrared light source 6 were collectively stored in a dedicated box. Thereafter, the still image or moving image stored in the computer 5 is projected on the screen. The visual sensor 3 is arranged on the same side as the projector 4 and in the vicinity thereof with the transparent column 1 as a boundary. When a human touches the transparent support column 1 from the outside of the transparent support column 1, that is, when a person outside the show window touches the image projected on the projection screen 2 through the glass, the visual sensor 3 Recognize shape.

  At this time, when the infrared light source 6 irradiates the projection screen 2 with infrared rays, noise from the periphery is reduced, and it becomes easy to detect the finger position. The infrared light source 6 is configured by arranging a large number of infrared light emitting diodes in a planar shape so as to be projected onto the projection screen 2 with uniform illuminance. Depending on the installation environment of the projection screen 2, infrared stray light may be strong. In that case, the detection noise may be reduced by increasing the output of the infrared light source 6. In addition, when the infrared stray light is weak, the output of the infrared light source 6 can be reduced to suppress power consumption.

The visual sensor 3 continuously obtains an image in which an image projected from the projector 4, a background projected from the outside of the transparent column 1, and an infrared projected image formed by irradiation from the infrared light source 6 are combined. This finger shape recognition calculation method is an arrangement of the method described in the embodiment, and includes a tenth step of performing start-up correction, and a step of capturing an image of the projection screen, recognizing and analyzing the finger shape. 20 steps, 30th step for determining whether the information meets the definition condition, 40th step for determining whether there is a finger position, and if there is the finger position, And a 60th step of sending the message clicked by the mouse to the loop of the window by the computer comparing the read finger position with the information transmitted from the visual sensor.

  The expert system is as described in the first embodiment and the embodiment.

  The serial port is as described in the first embodiment and the embodiment.

  FIG. 16 shows an embodiment in which the installation space can be further saved by projecting the image of the projector 4 through the reflection mirror 41. At this time, one or more reflecting mirrors may be used. The mirror surface does not necessarily have to be a flat surface, and may have a curved surface that corrects the distortion of the projection. The storage shown in FIG. 15 and FIG. 16 shows a state where it is installed on the floor surface, but it may be stored in a state where it is suspended from the ceiling using a ceiling hanger.

In FIG. 17, the projection screen 2 is sandwiched between the transparent column 1 and the transparent column 11.
1 and 11 are made of glass or synthetic resin which is transparent and transmits infrared rays. The projection screen 2 is made of glass or synthetic resin, but sometimes polarizing glass or liquid crystal is used.
As shown in FIG. 17, a liquid crystal is mounted between the glasses, and an electrode is attached to the liquid crystal, that is, the projection screen 2 to control the voltage, so that the projection light and external light of the projector are transmitted or scattered through the liquid crystal. It is something that can be done. The voltage control can be synchronized with the video / audio or the internal clock by the computer 5.

  As shown in FIG. 18, a projection screen that can be wound is disposed on the front surface of a transparent column. The transparent column 1 is mainly used for a show window, a full glass door, or the like. Images are projected only when a roll screen is used. The roll screen can be wound either electrically or manually.

  As shown in FIG. 19, the case where the transparent support | pillar 1 was a curved surface is shown. At this time, since the illuminance distribution of the infrared light source 6 changes, it is necessary to make correction so that the central portion does not become dark. For correcting the distortion of the projected image, it is preferable to use the projector 4 having a function capable of correcting the approximate value by correcting the angle of the divided screen in a plurality of portions. Approximate correction can also be set for straight lines, quadratic curves, and higher order curves. The projection screen 2 used at this time may be made of a soft material that can handle a curved surface.

INDUSTRIAL APPLICABILITY The present invention can be used for a system that sends instruction information to a computer by touching the projection screen with a finger and interacts with the computer and an operation method thereof.
In addition, the present invention, when a transparent door, window, or partition has a role of an internal / external boundary, looks at a screen image or image projected from the inside, and from the outside of the boundary onto the projection screen. By touching the finger, the instruction information can be sent to a computer inside the boundary to interact with the computer. At that time, even if the inside and outside of the boundary are completely blocked, it is possible to interact with the computer only through the screen of the projection screen. Since the wear and tear due to the security and installation environment of the installed equipment depends on the internal environment of the building itself, it is possible to construct and use a system that can be operated safely and for a long time.

1 is a conceptual diagram of a contact projection screen computer interaction system of the present invention. It is a block diagram of a visual sensor. 6 is a flowchart of a method of operation of a contact projection screen computer interaction system. It is explanatory drawing of a five-point localization. It is a block diagram of a digital signal processing circuit. It is a block diagram of a computer interface. It is a block diagram of a camera interface. It is a conceptual diagram of an expert system. It is a conceptual diagram of a neuron. It is explanatory drawing of a hidden layer. It is explanatory drawing of a neural network. It is a mathematical formula explaining a neural network. 3 is a mathematical formula for explaining a method of minimizing (maximizing) an error function E. 6 is an explanatory diagram of a method for minimizing (maximizing) an error function E. FIG. 1 is a conceptual diagram of an embodiment of a contact projection screen computer interaction system. FIG. It is a conceptual diagram of the Example using a reflective mirror. It is a side view which pinched | projected the projection screen between the transparent support | pillar and a transparent support | pillar. It is the side view which arranged the projection screen which can be wound up on the front surface of the transparent support | pillar. It is the top view which arranged the projection screen on the non-planar transparent support | pillar.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Transparent support | pillar 11 Transparent support | pillar 2 Projection screen 3 Visual sensor 31 CAMERA
32 CAMERA I / F
33 DSP
34 COMP ・ I / F
4 projector 41 reflection mirror 5 computer 6 infrared light source

Claims (19)

In a touch panel system including a transparent column, a projection screen, a projector, and a computer,
And a visual sensor comprising a digital camera, a camera interface, a digital signal processor and a computer interface,
The digital camera and the camera interface are connected, the camera interface is connected to the digital signal processor, and a signal output from the digital signal processor is connected to the computer by the computer interface,
The projection screen is installed on the transparent column, and the visual sensor is installed between the projection screen and the projector, or at the same position as the projector,
The touch sensor system, wherein the visual sensor and the projector are connected to the computer, respectively. The touch panel system according to claim 1, wherein the visual sensor is installed at the same height as the projection screen behind the projection screen .   The touch panel system according to claim 1, wherein the number of the digital cameras is one or more.   The touch panel system according to claim 1, wherein a material of the projection screen is polystyrene. The operation method of the touch panel system according to claim 1,
A first step of performing start-up correction;
A second step of picking up feature points of the projection screen and reducing the dimension;
A third step that the expert system sifts through,
A fourth step of determining whether the information meets the definition condition;
If there is relevant information, the fifth step proceeds to the neural network calculation method;
A sixth step of determining whether or not there is a finger position;
A seventh step of extracting the finger position if the finger position is present;
An eighth step of comparing the extracted finger position with information transmitted from the visual sensor;
9. A touch panel system operating method, comprising: a ninth step in which the computer performs a comparison process of information and performs control while driving a mouse. The startup correction method is a five-point localization method,
The computer cursor moves to the upper left corner-upper right corner-screen center point-lower left corner-lower right corner,
When the computer cursor stops at each position, the user moves the cursor to the next position with the laser pen, and the cursor moves to the next position.
6. The touch panel system operation method according to claim 5, wherein the operation proceeds to all five points while repeating the above operation at the next position. The expert system for determining finger information by judging finger information on a projected image includes a total database, a control strategy, and a generation formula rule.
The total database includes certain experience information and data held by the fingers recognized under image analysis recognized in practice, and some data information that can represent the finger information in the image during the image processing process And
Control strategy is a calculation method of finger information identification on how to reduce errors in various environments as much as possible,
6. The touch panel system operating method according to claim 5, wherein the generation formula rule is in charge of linkage and harmony between the total database and the control strategy.   6. The touch panel system operating method according to claim 5, wherein in the neural network calculation method, Mean Taxonomy is adopted as a hidden layer learning method, and a gradient descent method is applied as an output layer learning method. The projection screen is made of acrylic that can transmit infrared light,
The video camera is integrated with a digital video camera and an infrared light source that emits the infrared light,
The touch panel system according to claim 1, wherein the visual sensor is installed in the same direction as the projector. The touch panel system according to claim 1, wherein the visual sensor senses a shape of a user's finger on the projection screen, and a fingertip position is a mouse pointer position .   The touch panel system according to claim 9, wherein the infrared intensity from the infrared light source is adjusted according to an external infrared intensity. The operation method of the touch panel system according to claim 1,
A tenth step of performing start-up correction;
A twentieth step of capturing an image of the projection screen, recognizing and analyzing a finger shape;
A thirtieth step of determining whether or not the information satisfies a definition condition;
40th step of determining whether or not there is a finger position;
If there is the finger position, a 50th step of reading the finger position in the coordinates of the image;
A touch panel system operating method comprising: a 60th step in which a computer collates the read finger position with information transmitted from the visual sensor and sends a message clicked by a mouse to a window loop.   The touch panel system according to claim 1, wherein the transparent support is transparent glass.   The touch panel system according to claim 13, wherein at least one surface of the transparent support is diffused into a ground glass.   The touch panel system according to claim 1, further comprising a two-dimensional array of infrared light emitting diodes, wherein the projection screen is illuminated with infrared rays having uniform illuminance.   The touch panel system according to claim 1, further comprising a reflection mirror that reflects an image of the projector and projects the image on the projection screen. The projection screen is a liquid crystal with electrodes,
Sandwiching the liquid crystal with a transparent member,
The touch panel system according to claim 1, wherein the entire surface of the liquid crystal is made transmissive or scattering by an applied voltage to the electrodes.   The touch panel system according to claim 1, wherein the projection screen is rollable.   The touch panel system according to claim 1, wherein the projection screen is curved.

Images