JP3787649B2 - Pointing device - Google Patents

Description

[0001]
[Industrial application fields]
The present invention relates to a pointing device apparatus (hereinafter simply referred to as a pointing device), and in particular, there are one or more.regionThe user is watching fromregionIdentify and itsregionThe present invention relates to a pointing device in which an object corresponding to a gaze position is specified from one or more objects displayed on the basis of drawing data.
[0002]
[Prior art]
Conventional pointing devices include those discussed in Japanese Patent Application Laid-Open No. 60-85899, or IEICE Transactions A, J71-A, No. 12, pages 2192-2197. In this prior art, a user's gazing point is obtained using an eye camera or an eye camera and a field of view camera, and can be used for a pointing device on a screen. As another prior art, for example, in the one disclosed in Japanese Patent Laid-Open No. 62-196723, a point light source is provided at the periphery of the display screen, and based on an image of the point light source reflected in the field of view camera, the user's The rotation angle of the head is detected, and the position of the eyeball detected by the eye camera is corrected. In addition, in the devices disclosed in Japanese Patent Laid-Open Nos. 62-53630 and 62-53631, an eye camera is installed based on the position of the feature in the field-of-view image by installing the feature in the measurement space. Correct the viewpoint information from. In these inventions, the viewpoint information can be corrected even when the user position is moved. Further, a technique described in Japanese Patent Application No. 60-99532 is known as a technique for using a pointing signal for pointing, and the user can perform pointing while moving.
[0003]
The above-mentioned prior arts disclosed in Japanese Patent Application Laid-Open No. Sho 60-85899 and the Journal of the Institute of Electronics, Information and Communication Engineers, and those shown in Japanese Patent Application Laid-Open No. Sho 62-196723 are subject to movement of the eyeball and head. Thus, it is effective for obtaining the viewpoint position of the user, but cannot be used when the position of the user's body moves greatly. JP-A-62-253630 and JP-A-62-253631 store the initial visual field image and the previous visual field image, and based on this and the latest visual field image, the user's head position is stored. Therefore, there is a problem that the process becomes complicated when the user's movement is large and the feature is temporarily deviated from the visual field image. Furthermore, when there are a plurality of display screens, no consideration is given to specifying which part of which display screen the viewpoint is on. Further, the device disclosed in Japanese Patent Application No. 60-99532 has a problem that it is necessary to arrange a large number of light receiving elements around the screen, and in particular, a large display screen cannot be easily manufactured.
[0004]
The object of the present invention is to provide one or moreregionOne or more objects are displayed on eachOr existFirst, the user is watchingAn area can be specified, and an object corresponding to the gaze position can be easily specified from the area using drawing data.pointing deviceapparatusIs to provide.
[0005]
[Means for Solving the Problems]
In order to achieve the above object, the present invention includes one or more.regionThe user is watching fromregionIdentify and itsregionA pointing device device that identifies an object corresponding to a gaze position from one or more objects that are displayed based on drawing data within the pointing device.AreaThe user is gazing based on identification means for individually identifying, a field camera and an eye camera worn by the user, and a viewpoint position of the user in the monitor screen of the field camera and the position of the identification means Identify the area,The coordinates of the viewpoint position of the user in the area where the user is gazing is specified by the position of the identification means and the coordinates of the viewpoint position of the user in the monitor screen of the visual field camera,The user is watchingregionAnd a means for specifying an object being watched by the user based on the viewpoint position of the user and the coordinate position of the drawing data.
[0006]
[Action]
Even if the user moves or there are a plurality of areas, the present invention easily identifies which area the user is gazing by using the user's viewpoint position and identification means in the monitor screen, It is possible to specify using drawing data whether one or more objects in the region being watched are being watched..
[0007]
【Example】
Hereinafter, the present invention will be described with reference to examples. FIG. 1 is a block diagram showing an embodiment of the present invention, and infrared lamps 1101 are provided at the four corners around the display screen 1001 of the display device 11. On the display screen 1001, the drawing of the display drawing data 1301 is displayed via the display control device 1201. A user 2401 is equipped with an eye camera 1401 that detects the movement of the eyeball, a small-sized field of view camera (sensitive to infrared rays) 1501, and a microphone 1901 for voice input to capture the front view of the user is doing. A keyboard 1801 is prepared as another input device. The camera signal processing device 1601 obtains the position coordinates of the viewpoint in the display screen 1001 based on the signals from the eye camera 1401 and the field camera 1501. Based on this signal, the object specifying device 1701 specifies the object in the display screen that the user is watching from the display screen data being displayed on the display screen 1001. On the other hand, the input analysis device 2001 analyzes input from the microphone 1901 and the keyboard 1801 using the input analysis data 2101. This result is input to the input signal integrated processing device 2201, and is output to the computer 2301 as a processing command together with information on the watched object from the object specifying device 1701.
[0008]
FIG. 2 is a flowchart showing the operation of the camera signal processing device 1601. In this apparatus, this process is always performed periodically. First, the signals from the field camera 1501 and the eye camera 1401 are captured (step 9111), and then the position coordinates of the infrared lamp in the field of view based on the luminance level from the field camera 1501 signal, that is, the monitor screen of the field camera. Is obtained (step 9112). Next, the position coordinates of the viewpoint in the monitor screen of the visual field camera 1501 are obtained from the signal of the eye camera 1401 (step 9113). Furthermore, based on the information such as the infrared lamp position on the actual display screen from the position coordinates and the viewpoint position coordinates of the infrared lamp in the monitor screen, a normal image processing technique such as perspective transformation is used to display the display screen 1001. The coordinates of the viewpoint are obtained (step 9114).
[0009]
FIG. 3 is a flowchart showing the operation of the object specifying device 1701. First, the coordinates of the viewpoint detected by the camera signal processing device 1601 are captured (step 9121), and the object displayed at the viewpoint position is identified by referring to the display screen data based on the coordinates of the viewpoint (step 9121). 9122).
[0010]
FIG. 4 is a diagram showing an example of the display drawing data 1301 and includes a name of a display object and coordinates on the display screen 1001 on which the display object is displayed. However, the coordinates are the coordinates of each vertex when the area where the display object is displayed is approximated by a rectangle. It is also possible to approximate the region with a polygon. In this case, it is necessary to prepare coordinate data corresponding to the number of each vertex.
[0011]
Next, operations of the input analysis device 2001 and the input signal comprehensive processing device 2201 will be described. These devices operate only when there is input from a microphone or keyboard.
[0012]
FIG. 5 is a flowchart showing the operation of the input analysis apparatus 2001. This is to analyze an input sentence from the keyboard 1801 or the microphone 1901 and convert it into a processing instruction, that is, a function and an argument value, and output it. As an input sentence, “Attack trend display” is input in step 101. The operation will be described by taking the case as an example. The morpheme / syntax analysis of the input sentence is performed with reference to the part-of-speech category of words and the grammar 201 described in the dictionary 202. As a result, the input sentence is divided and written, and a semantic category and meaning are added to each word. Here, as a morpheme / syntactic analysis method, a commonly used method such as a prediction bottom-up method is used. Grammar is a rule about sentence structure.
S (sentence) → NP (noun phrase) S (sentence)
Represents the rule that S on the left side can be composed of NP and S on the right side. In addition, the dictionary provides four pieces of information for each headword: a part of speech category, a display word obtained by converting the headword into Kana-Kanji, a semantic category, and a meaning. However, as for particles, only two parts are given: part of speech category and display word. In addition, the recognition of the word contained in an input sentence can be similarly implemented also by the dictionary lookup by the longest match method normally used with a word processor etc.
[0013]
Next, the processing step 103 for identifying the instruction content is executed. In this process, the result of the syntax analysis is taken in and the instruction content is identified using the instruction identification rule 203. Here, the command is, for example, SHOW-SVAR (state quantity presentation command). The instruction identification rule gives a standard sentence pattern for identifying each instruction to each instruction. In the instruction identification, as will be described in detail with reference to FIG. 6, it is assumed that the matching between the input sentence and each sentence pattern is tried sequentially, and the instruction corresponding to the matched sentence pattern is the input instruction. In the example of the figure, the sentence pattern “¥ <state quantity>? Presentation + seyo” matches the input sentence, and the identification result that the input instruction is SHOW-SVAR is obtained. In this process, several symbols are prepared to give sentence patterns. “%” Is a symbol representing a word used for matching only when there is a particle in the input sentence, and can correspond to omission of a particle often seen in a dialogue sentence. “+” Is a symbol that represents an optional word, and is used to cope with body stop. “?” Is a symbol representing an arbitrary number of words, which makes it easy to set a sentence pattern. Further, “¥” represents matching with a semantic category of a word.
[0014]
Here, the detailed processing contents of the instruction content identification unit will be described. FIG. 6 is a flowchart of the process. First, the sentence pattern number is set to 1. For the sentence pattern, i indicating the word number in the sentence pattern is set to the total number I of words in the sentence pattern. Further, j indicating the word number of the input sentence is set to the total number J of words of the input sentence (step 401). Next, the i-th word Pi of the k-th sentence pattern and the j-th word Wj of the input sentence are brought (step 402). A matching process described below is performed on Pi and Wj. Since i is initially set to I and j is set to J, words are compared from the end of the input sentence and the sentence pattern toward the beginning of the sentence.
[0015]
The matching process differs depending on the shape of the word Pi of the sentence pattern, that is, the presence or absence of the aforementioned symbol (step 403). First, processing when no symbol is attached will be described. In this case, Pi and Wj are compared (step 404), and if they are the same, i and j are reduced by 1 (step 405). That is, the word to be compared is the previous one. When this result is 0 (step 406), since all the words of the sentence pattern match the words of the input sentence, it is assumed that the instruction name corresponding to the sentence pattern k is the input instruction content. Output (step 410).
[0016]
On the other hand, if i is 1 or more (step 406), the determination for j is performed (step 407). As a result, if j is 1 or more, the processing from step 402 is repeated. Conversely, if j is 0, matching between the sentence pattern k and the input sentence has failed, and preparation for matching with the next sentence pattern (k + 1) is made (step 408). If the sentence pattern still remains (step 409), the processing from step 402 is executed for the new sentence pattern. On the other hand, when there are no more sentence patterns, a message indicating that the instruction cannot be identified is output and the processing is terminated (step 412).
[0017]
If the symbol “¥” is attached to Pi, the semantic categories of Pi and Wj are compared (step 413). Depending on the result, the same processing as described above is performed.
[0018]
If the symbol “%” is attached to Pi, it is determined whether or not the part of speech category of Wj is a particle (step 414). As a result, if it is a particle, Pi and Wj are compared (step 415), and processing similar to that described above is performed according to the result. On the other hand, if Wj is not a particle, j is reduced by 1, and if j is 1 or more (step 417), the process returns to step 402. On the other hand, if j is 0, there are no more words in the input sentence, so that matching between the sentence pattern k and the input fails, and the process for the next sentence pattern, that is, the process from step 408 is performed.
[0019]
If “Pi” has a symbol “+”, Pi and Wj are compared (step 418). If they are the same, the processing from step 405 is executed. On the other hand, if they are different, after i is decreased by 1, if i is 1 or more, the process returns to step 402. When i becomes 0, it means that all the words of the sentence pattern k are included in the input sentence (except for those that can be omitted), so the process returns to step 410 and ends the instruction identification.
[0020]
If Pi is the symbol “?”, The words Pi−1 and Wj before “?” Are compared (step 421). If the word matches as a result, 1 is subtracted from i, and the matching process for the next word is executed from step 405. On the other hand, if j is not the same, j is decremented by 1 and if j is 1 or more (step 424), matching processing is performed using the word immediately before the input sentence from step 402. If j is 0, the matching with the sentence pattern k has failed, so the matching process with the next sentence pattern k + 1, that is, from step 408 is executed.
[0021]
When the instruction identification process in step 103 of FIG. 5 detailed above is completed, a process for acquiring information necessary for executing the instruction identified in step 104 is performed. That is, since some information is required for execution depending on the instruction, necessary information items and the acquisition method are prepared as frame format data as a method for controlling the acquisition of the information. This is called a case frame. The case frame is defined for each instruction, and an item of information necessary for executing the instruction is given to each slot of the instruction name frame. In the example of the case frame 204 in FIG. 5, it is indicated that three pieces of information “state quantity”, “display form”, and “location” are necessary for the instruction SHOW-SVAR. The data and data necessary to obtain the information are given to the face and value of each slot. The semantic category name of the word to be searched from the input sentence is given to the VALUE facade, and the name of the function to be activated when a necessary word is not present in the input sentence is given to the IF-NEEDED face. Here, the function STACK is a function that refers to the conversation history stack, and the function ASK is a function that requests an additional input of information from the questioner. The function CAMERA is for using the result from the object specifying device 1701. When this function is invoked, the corresponding case value is? Write a Place.
[0022]
The process flow of step 104 is shown in the flowchart of FIG. First, a case frame having an instruction name as a frame name is extracted (step 501). Next, i is set to 1 (step 502). After this initial processing, the value of the VALUE fact in the i-th slot is read and set as the value of Vi (step 503). If the value of Vi is not nil (step 504), the input sentence is searched for a word having a semantic category (step 505). If the corresponding word can be found (step 506), the obtained word is set as the value of the information item of the i-th slot (step 507). If there are still slots (step 510), i is increased by 1 ( Step 511), and moves to the processing for the next slot. On the other hand, if the word cannot be found in step 506, or if the value of Vi is nil in step 504, the value of the IF-NEEDED face is read and set as the value of Ni (step 508). Next, Ni functions are sequentially executed. As a result, if a value is obtained, it is set as the value of the information item of the i-th slot (step 509), and the process proceeds to the next slot.
[0023]
The above operation will be described with reference to the input example of FIG. 5. The case frame corresponding to the instruction SHOW-SVAR has three slots of “state quantity”, “display form”, and “location”. As for “state quantity”, first, a word having a semantic category <state quantity> given as a value of the VALUE facilit is searched in the input sentence. The input sentence now includes the corresponding word “pressure”. Thereby, “pressure” is set as the value of “state quantity”. Similarly, regarding the display form, “trend” is obtained as a word having a semantic category <display form> from the input sentence and set as a value. As for “place”, words in the semantic category <Place> are searched from the input sentence, but there is no corresponding word in the input sentence. Therefore, the value of the IF-NEEDED face is referred to and the function CAMERA is executed. As a result, “? Place” is set as the value of “place”.
[0024]
When the acquisition of information necessary for instruction execution is completed as described above, the process for associating with the function / argument in step 105 is finally executed. In this process, the value of the instruction and information item obtained in the previous process is output in association with the function and argument of the program. In the input example of FIG. 5, the function name is set to “trend display” from SHOW-SVAR and the “display form” trend. In addition, “pressure” and “? Place” are set as argument values.
[0025]
FIG. 8 is a flowchart showing the operation of the integrated input signal processing apparatus 2201. In this apparatus, first, a function name and an argument value determined by the input analysis apparatus 2001 are read (step 9151). Then the variable in the argument? It is determined whether or not there is a place (step 9152). As a result, if there is no variable, the function and argument values from the input analysis device 2001 are output to the computer as processing instructions (step 9153). On the other hand, if there is a variable, the object being watched by the user is read from the object specifying device 1701 (step 9154). Furthermore, variables? The name of the object is set in Place, and the processing instruction is completed and output to the computer (step 9155).
[0026]
As described above, if the apparatus according to the present embodiment is used, a signal about the viewpoint from the eye camera and the field-of-view camera worn by the user and a signal about the image of the infrared lamp provided in the peripheral portion of the display screen are transmitted. It is possible to obtain the coordinates in the display screen that the user is gazing at and the object displayed there. This makes it possible to perform pointing that is simple and does not limit the operation location. In addition, voice, input from the keyboard, and information on the object to be watched can be integrated into a command to the computer. As a result, not only the pointing operation but also simplification of the input work to the computer and improvement of workability can be realized.
[0027]
In the embodiment described above, infrared lamps are provided at the four corners of the display screen as marks for giving the shape of the display screen. However, various modifications can be made. FIG. 9 shows an example. Infrared light is sent from the infrared light emitting sources 2501 to 2504 to the four leaking fiber frames 2701 to 2704. Further, each infrared light emitting source blinks, and the blinking period varies depending on each light emitting source. Yes. When such a mark is used, the camera signal processing device 1601 takes a viewpoint from the mark and viewpoint signals in the image of the field-of-view camera to determine which mark the frame is surrounded by. Obtained from the blinking cycle of the clogging mark. Thereafter, if the coordinates of the viewpoint within the specified frame are obtained, the coordinates of the viewpoint on the display screen can be obtained with high accuracy. This is particularly effective when a display device such as a large screen display device is used.
[0028]
FIG. 10 is a schematic diagram showing another example of the mark. Infrared rays from one infrared light emitting source 2505 are guided to point light sources 2801 to 2804 via a star coupler 2601, a leakage fiber 2700, and an optical fiber. The display screen 1001 is divided into four parts, each of which has a different number of point light sources. When this mark is used, it is determined in which divided region the viewpoint is located from the light from the leaking fiber and the number of point light sources in the same divided region as the viewpoint. Other processes are the same as in the example of FIG. According to this example, the coordinate accuracy of the viewpoint can be improved and the number of infrared light sources used can be one.
[0029]
In these examples, the display screen 1001 is divided into four by marks, but the number of divisions can be changed. Further, instead of marking all the display screens, it is possible to mark only a part of the display screen and use only that part as an area that can be pointed. For example, a part of the screen can be surrounded by a leaking fiber so that voice input is accepted only when the viewpoint enters this part, and can be used for switching voice input. Although an infrared light source is considered here as a light source, the same function can be realized by using a light source of another wavelength, for example, a visible light source.
[0030]
FIG. 11 is a block diagram showing a second embodiment of the present invention. Three display screens 1001 to 1003 are installed on the display panel 2901, and red, green and blue marks 3001 to 3003 are provided on the periphery thereof. Is attached. On each display screen, the drawing of the display drawing data 1302 is displayed by the display control device 1202. A user 2401 wears an eye camera 1401, field-of-view cameras 1501 and 1502, a voice input microphone 1901, and a keyboard 1801. The camera signal processing device 1602 determines which coordinates on which display screen the user's viewpoint is based on the signals of the eye camera 1401 and the field of view cameras 1501 and 1502. Based on this signal, the target object specifying device 1702 obtains the target object being watched by the user in the drawings displayed on the display screen. On the other hand, the input analysis device 2001 analyzes input from the microphone 1901 and the keyboard 1801 using the input analysis data 2101. This result is input to the input signal general processing unit 2202. Next, the target object and the target limitation data 2203 are identified by this apparatus using the target object limitation data 2203, and the processing instruction is output to the computer 2301 in combination with the signal from the input analysis apparatus 2002. . The processing of each of these units is roughly the same as in the case of FIG. 1, but there is a difference in that point because the number of display screens and marks are different. Details thereof will be described below.
[0031]
FIG. 12 is a flowchart showing the operation of the camera signal processing device 1602. First, signals from the field-of-view cameras 1501 and 1502 and the eye camera 1401 are captured (step 9211), and then the position coordinates of the viewpoint are obtained on the monitor screen of the field-of-view camera (step 9212). Next, the marks 3001 to 3003 for which the viewpoint is set on the monitor screen of the field-of-view camera are specified, and the color is discriminated (step 9213). For example, when the mark color is recognized to be red, it is specified that the viewpoint is in the display screen 1001. Next, for the determined mark, the coordinates of the four corner points of the mark within the monitor screen of the visual field camera are obtained (step 9214). From the position coordinates and the position coordinates of the viewpoint, information on the actual mark shape is used to obtain the coordinates of the viewpoint on the display screen (step 9215). This processing is performed by a normal image processing method.
[0032]
Next, the object candidate specifying device 1702 obtains a candidate for the object being watched by the user from the data regarding the drawing displayed on the display screen specified as having a viewpoint. The processing here is the same as the processing in the object specifying device 1701 in FIG. 1 described above. However, the contents of the display drawing data are different, and a plurality of objects are allowed to be associated with points in the same screen. That is, the target object specifying device outputs one or more target object candidates from the coordinates of the viewpoint. In the following, it is assumed that a drawing of a pressure vessel of a nuclear power plant is displayed on the display screen, and two steam-water separators and a steam dome are obtained as object candidates.
[0033]
Next, the input from the microphone 1901 or the keyboard 1801 is analyzed by the input analysis device 2001. This performs exactly the same processing as in FIG. So now from Mike 1901
“Show the pressure trend”
When this input is considered, this input is analyzed,
Trend display (pressure,? Place)
Is identified. here,? Place is a variable representing a place. The input command comprehensive processing device 2202 takes in the identified command and the candidate of the object previously obtained, and creates a command for the computer 2301.
[0034]
FIG. 13 is a flowchart showing the operation of the input signal general processing unit 2202. This device first reads a function name and an argument value from the input analysis device 2001 (step 9221). Then the variable in the argument? It is determined whether or not there is a place (step 9222). As a result of judgment, variable? If there is no Place, the function and argument from the input analysis device 2001 are output as they are as processing instructions to the computer 2301 (Step 9223). On the other hand, if there is a variable, the object candidate being watched by the user is read from the object candidate specifying device 1702 (step 9224). In this case, two steam / water separators and steam domes are obtained as object candidates. Next, the object limited data is searched based on the value of a specific argument (in this case, the state quantity, the value is “pressure”) (step 9225).
[0035]
Here, the object limitation data 2203 will be described. FIG. 14 is a diagram showing the contents of a part of the object limitation data, and the relationship between the state quantity and the parts in the plant is given. This data indicates that, for example, pressure is measured for a dry well, a steam dome, and the flow rate is measured for a core and the like.
[0036]
Returning to FIG. 13, after this data is searched in step 9225, it is determined whether or not there is an object candidate corresponding to the argument value (“pressure” in this example) (step 6226). As a result, if not, the user is requested to input the location (step 6229). According to this request, if there is an input from the user with a microphone or the like, it is analyzed and a processing instruction is completed. On the other hand, if there is a candidate, it is determined whether or not there is one (step 6227). As a result, if there are two or more candidates, the user is requested to select (step 6230). If there is one candidate, is it a variable? Set as the value of Place, complete the instruction and output it to the computer (step 6228). In the example we are considering, we obtained two steam / water separators and steam domes as the object candidates. However, the object-limited data is obtained from the argument value “pressure” obtained by analyzing the input from the microphone. By reference, the steam dome, which is the part where the pressure is measured, is selected. As a result, the instruction
Trend display (pressure, steam dome)
Is output to the computer 2301.
[0037]
As described above, when the apparatus according to the present embodiment is used, when a plurality of display screens are used, it is possible to specify which part of which display screen the user's viewpoint is on. Therefore, it is effective for command input in the case of plant operation using a large number of display screens. In the present embodiment, an example in which the number of users is one is shown, but the same processing can be performed when there are a plurality of users.
[0038]
In the embodiment of FIG. 11, the display screens 1001 to 1003 are surrounded by different color frames for identification. FIG. 15 shows an example in which this is performed by another method. In the figure, a display board 2901 is provided with three display screens 1001 to 1003, and a mark 3004 is provided around each of them. This mark is written with an infrared or visible light source or paint such as paint. In this way, the display screen can be distinguished by the number of marks. Further, by changing the number of marks on the left and right and top and bottom of the display screen, it is possible to recognize which part of which display screen is reflected in the image of the visual field camera. Further, not only the number and arrangement of marks, but also the color, light source intensity, blinking cycle, and combinations thereof can be distinguished. By using such a mark, it is possible to easily determine the top, bottom, left, and right of the screen even when the user's head is greatly rotated.
[0039]
FIG. 16 is a block diagram showing the configuration of the apparatus according to the third embodiment of the present invention. An arrow mark 3302 is attached to the hat 3301 of the user 2401, and the monitor camera 3101 monitors this. The display screen specifying device 3201 specifies a display screen in the direction in which the user is facing based on the signal of the arrow mark 3302 captured by the monitor camera 3101. The camera signal processing device 1603 obtains the viewpoint in the display screen specified by the above processing based on the signals of the eye camera 1401 and the field of view camera 1501 worn by the user, and inputs them to the computer 2301.
[0040]
Here, the display screens 1001 to 1003 are not marked on the periphery or inside thereof, and the display screen uses the shape and color of the frame as marks. According to this method, it is not necessary to mark the display screen or the inside thereof, and a pre-existing one can be used as a mark, and labor for creating a display request or the like can be saved. Further, the user's movement can be monitored by monitoring a hat arrow to narrow down the display screen candidates that the user is viewing. This method can be used in combination with the second embodiment, and is particularly useful for speeding up and simplifying the specific processing when the number of display screens is large.
[0041]
Here, the shape and color of the frame of the display screen are used as marks. However, if the image processing technology advances, the object displayed on the part where the user's viewpoint is recognized by recognizing the drawing displayed on the display screen. It is possible to directly identify things.
[0042]
FIG. 17 shows a fourth embodiment of the present invention. In the figure, meters 3401 to 3404 and switches 3601 to 3604 are installed on a control panel 3501 of the plant. Light from the infrared light sources 2506 to 2512 is guided to branching devices 3701 to 3707 by optical fibers 3801 to 3807. Leakage fibers 3901 to 3907 are connected to this branching device, and this fiber is arranged so as to surround a switch and a meter. The infrared light emitting sources 2506 to 2512 generate light having different wavelengths.
[0043]
On the other hand, the eye camera 1401 and the field-of-view camera 1501 worn by the user 2401 are taken into the computer 4201 via the data input unit 4202 via the signal transmission device 4001 and the signal reception device 4101. This signal is stored in the storage device 4401 via the device specifying unit 4601, the data recording unit 4602, and the output unit 4203. The device specifying unit 4601 uses the device specifying data 4402 via the input unit 4207 and the output unit 4208. On the other hand, when a command is input from the command input device 4301 via the input unit 4205, the data search unit 4603 operates, and the recorded data is read from the storage device 4401 via the input unit 4202, and the input command Are displayed, printed, or copied on the output device 4501 via the output unit 4206.
[0044]
This embodiment is for automatically recording and processing, for example, the movement of the operator's viewpoint during plant operation. When activated, the processing of FIG. 18 is periodically performed. First, it is specified which of the leaked fiber frames captured by the field-of-view camera (infrared camera) contains the viewpoint from the eye camera. A detailed flowchart of this processing step 9301 is shown in FIG. 19, which is executed by the device specifying unit 4601. That is, first, signals from the field-of-view camera and eye camera are captured (step 9131), and the position coordinates of the viewpoint on the monitor screen of the field-of-view camera are obtained from the eye camera signal (step 9132). Next, a frame for capturing the viewpoint is specified on the monitor screen of the field-of-view camera (step 9133). Next, the blinking cycle of the infrared light source of the frame is obtained from the luminance information of the specified frame (step 9134). Based on this blinking cycle, the number of devices in the frame is obtained using the device specifying data (step 9315). As a result, when there is only one device within the specified frame (step 9316), the device is specified and the process is terminated (step 9317). On the other hand, when there are a plurality of devices in the frame, the coordinates of the viewpoint in the frame are obtained by the same processing as the camera signal processing device 1602 in the embodiment of FIG. 11 (step 9318). Next, the device is specified from the coordinates of the viewpoint using the device specifying data (step 9319).
[0045]
A part of the device specifying data used here is shown in FIG. This data gives the number of devices in the frame and the device name for each blinking cycle of the infrared light source of the frame. Further, when there are a plurality of devices in the frame, each region is approximated by a rectangle for defining a region corresponding to each device, and the coordinates of four vertices of the rectangle are given. Here, when defining each region, it is possible to provide only the x coordinate of the dividing point when dividing the frame into two left and right regions, or to define the region by a polygon, a circle, or the like.
[0046]
As described above, when the display screen in which the viewpoint is entered is specified in step 9301 in FIG. 18, the time at that time and information related to the name of the meter, switch, etc. that the operator is viewing are displayed next. And stored in the storage device 4401 (step 9302).
[0047]
On the other hand, when a display command is input from the command input device 4301, the processing of FIG. 21 is started (step 9303), and information about the temporal operation of the operator's viewpoint stored in the storage device 4401 is output to the output device. Displayed in 4501 or output in another form (step 9304). The output to the output device can be output in parallel with the data processing, that is, as a monitor if there is a request from the command input device. In addition, in the apparatus which becomes a present Example, the apparatus etc. which the operator watched were specified by arrange | positioning a leak fiber and changing each blinking period. This identification can also be performed by various methods shown in the above-described embodiments. Alternatively, it can be realized in the same manner by recognizing the shape of the switch or meter, or the letters on the nameplate installed therewith.
[0048]
According to the present embodiment, the movement of the viewpoint of the operator during plant operation can be monitored and recorded together with the name or number of the object being watched. Therefore, for example, the frequency of looking at the meter, the viewpoint operating range within a certain time, etc. can be easily analyzed using a computer, which can be utilized for the evaluation and design of the control panel. Although only the viewpoint movement is monitored in this apparatus, the utterance history of the operator can be recorded and used in the same manner by using a microphone and a voice recognition apparatus.
[0049]
FIG. 22 is a block diagram showing a fifth embodiment of the present invention, and a leakage fiber 3908 connected with an infrared light source 2513 is arranged around the display screen 1001. A user 2401 has an infrared projector (pointer laser) 4701 or wears it on a finger or the like. The light projector is provided with a light on / off switch 4702. The monitor infrared camera 3102 is fixedly installed in front of the display screen 1001, and its signal is input to the computer 2301 via the pointing position specifying device 5001. Further, the pointing position input to the computer is displayed on the display screen 1001 as a pointing marker 4901 via the display control device 1204.
[0050]
FIG. 23 is a flowchart showing an outline of processing in the apparatus according to the present embodiment. First, in step 9401, the position of the point on the display screen where the laser light of the finger pointing laser collides and reflects on the display screen is obtained from the monitor camera signal by the finger position specifying device 5001. The detailed contents of this step 9401 are shown in FIG. 24. First, the signal of the monitor camera 3102 is read (step 2411), the bending point of the pointing laser beam from the projector 4701 is detected from this signal, and the point of that point is detected. The coordinates in the monitor screen are obtained (step 9412). Next, a straight line expression representing each side of the frame made of the leaking fiber 3908 is obtained on the monitor screen coordinates (step 9413). Next, from the equation of each side of the frame and the coordinates of the bending point, the coordinates on the display screen of the bending point, that is, the point pointed by the user, are obtained using information about the actual frame shape (step 9414). This processing is performed by a normal image processing method. In this embodiment, a frame is formed around the display screen by an infrared light source and a leaking fiber. However, since the monitor camera can be fixed, a similar finger point identification process can be performed even when there is no frame if a relationship necessary for conversion between coordinates on the monitor screen and coordinates on the display screen is prepared in advance.
[0051]
Next, returning to FIG. 23, the coordinates obtained in step 9401 are input to the computer 2301 and used as pointing information by pointing (step 9402). Further, the coordinates obtained in step 9401 are input to the display control device 1204 and displayed on the display screen (step 9403).
[0052]
As described above, according to the apparatus of the present embodiment, the finger signal can be used for pointing. In contrast to the conventional technology that requires a large number of light receiving elements to be arranged around the screen, the device can be configured with a monitor camera, an infrared projector, and an infrared light source and a leakage fiber, which makes it easy to enlarge the screen. Yes. Further, in this embodiment, the case where there is one display screen is shown, but even when there are a plurality of display screens, as shown in the other embodiments, the flashing cycle of the infrared light source is changed for each display screen. It can be realized in the same manner by a method such as using the mark, using a paint mark, or using the shape of the display screen as a mark.
[0053]
【The invention's effect】
As described above, according to the present invention, even when the user moves, the information about the movement of the user's eyeball can be used for pointing the display screen, and further for pointing the target object within the specific display screen. Because of the non-contact operation, a pointing device that does not limit the operation location can be realized.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.
FIG. 2 is an operation explanatory diagram of the first embodiment.
FIG. 3 is an operation explanatory diagram of the first embodiment.
FIG. 4 is an operation explanatory diagram of the first embodiment.
FIG. 5 is an operation explanatory diagram of the first embodiment.
FIG. 6 is an operation explanatory diagram of the first embodiment.
FIG. 7 is an operation explanatory diagram of the first embodiment.
FIG. 8 is an operation explanatory diagram of the first embodiment.
FIG. 9 is a diagram illustrating a method for identifying a display screen.
FIG. 10 is a diagram illustrating a method for identifying a display screen.
FIG. 11 is a block diagram showing a configuration of a second exemplary embodiment of the present invention.
FIG. 12 is an operation explanatory diagram of the second embodiment.
FIG. 13 is an operation explanatory diagram of the second embodiment.
FIG. 14 is an operation explanatory diagram of the second embodiment.
FIG. 15 is a diagram showing a method for identifying a display screen in the second embodiment.
FIG. 16 is a block diagram showing a configuration of a third exemplary embodiment of the present invention.
FIG. 17 is a diagram showing a configuration of a fourth exemplary embodiment of the present invention.
FIG. 18 is an operation explanatory diagram of the fourth embodiment.
FIG. 19 is an operation explanatory diagram of the fourth embodiment.
FIG. 20 is an explanatory diagram of the operation of the fourth embodiment.
FIG. 21 is an operation explanatory diagram of the fourth embodiment.
FIG. 22 is a block diagram showing a configuration of a fifth exemplary embodiment of the present invention.
FIG. 23 is an operation explanatory diagram of the fifth embodiment.
FIG. 24 is an operation explanatory diagram of the fifth embodiment.
[Explanation of symbols]
1001 to 1003 ... Display screen, 1101 ... Infrared lamp, 1301 and 1302 ... Display drawing data, 1401 ... Eye camera, 1501 and 1502 ... Field of view camera, 1601 to 1603 ... Camera signal processing device, 1701 ... Object identification device, 1702 ... Object candidate identification device, 1801 ... keyboard, 1901 ... microphone, 2001 ... input analysis device, 2101 ... input analysis data, 2201, 2202 ... input signal comprehensive processing device, 2203 ... object limitation data, 2301 ... computer, 2401 ... User, 2700-2704, 3901-3907 ... Leaked fiber, 3001-3004 ... Mark, 3101,3102 ... Monitor camera, 3201 ... Display screen identification device, 3301 ... Cap, 3302 ... Arrow mark, 4201 ... Computer, 4301 ... Command input Device, 4601 ... device specifying unit, 4701 ... infrared projector, 4901 ... marker, 5001 ... finger pointing position specifying device.

Claims (4)

A pointing device device that identifies a region in which a user is gazing from among one or more regions, and identifies a target corresponding to a gaze position from one or more objects in the region. ,
An identification means for individually identifying one or more areas, comprising a mark formed by color around the area, a mark emitting light around the area, or a mark using the shape of the area itself ;
A field-of-view camera that is worn by the user and captures the range that the user is viewing;
An eye camera that is mounted on the user and detects the user's viewpoint position within the screen shot by the field-of-view camera;
Specify the region in which the user is gazing in the shooting screen of the visual field camera by the viewpoint position detected by the identification means and the eye camera ,
The coordinates of the viewpoint position of the user in the area where the user is gazing are specified by the position of the identification means and the coordinates of the viewpoint position of the user in the shooting screen of the visual field camera,
The user of the viewpoint position and drawing data even properly based on the coordinate position of the device specific data in the region where the user is gazing, means for identifying an object that the user is gazing,
A pointing device device comprising: The pointing device apparatus according to claim 1, wherein the viewpoint position of the user in the specified area is displayed by a marker on a display screen .   3. The pointing device apparatus according to claim 1, wherein the identification unit identifies one or more regions by changing a color or number of marks, a light emission wavelength, or a blinking cycle.   3. The pointing device apparatus according to claim 1, wherein the identification unit identifies one or more areas by being a specific graphic attached to the area or a mark attached to the specific graphic. .

Images