JP2003122505A - Information processor, information processing system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable the operation similar to that of a touch panel even if a display screen does not adopt a structure of a touch panel. SOLUTION: A point position (actual coordinates) which is a position on the display screen designated by a pointer is recognized from a pickup image obtained by imaging the display screen with a camera part, and according to the recognized point position, a required information processing responding to GUI operation is performed. Contact sound generated by a physical pointer's contact with the display screen is collected by a microphone to perform analysis processing, and according to the result of the analysis processing, a required information processing responding to GUI operation is performed. Thus, the pointer is brought into contact with the display screen to thereby perform the operation corresponding to the mouse movement and the operation corresponding to the left button of the mouse.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing device, an information processing system, and a program,
In particular, the present invention relates to an information processing device and an information processing system that execute required information processing in accordance with an operation performed on a graphical user interface, and a program to be executed by these devices and systems.

[0002]

2. Description of the Related Art Conventionally, a GUI displayed on a display screen
It is widely known that a touch panel is provided on a display screen so that an image can be operated by directly touching the screen with a finger or a pointer pen.
As such a touch panel, a structure example of a so-called pressure-sensitive touch panel is briefly shown in FIG. FIG. 15 (a)
As shown in (b), the pressure-sensitive touch panel has a structure in which an upper electrode 101 and a lower electrode 102 are arranged so as to face each other via a spacer 103. Lower electrode 10
2 is configured by forming a silver electrode on a glass substrate, for example, and the upper electrode 101 is configured by forming a resistance film on a PET film or the like. Conductive wires are drawn out from the silver electrode and the resistance film, respectively. Then, when a certain position on the upper electrode 101 side is pressed by, for example, a user's finger or a pointer pen, the resistance film of the upper electrode 101 comes into contact with the silver electrode of the lower electrode 102 due to the bending of the PET film caused by this pressing force. To do. The operation position is detected by detecting the resistance value or the like that changes according to the contact position.

[0003]

By the way, for example, a GUI in an information processing device such as a personal computer.
As a well-known operation, a pointing device typified by a mouse is used for the operation. However, in such an operation using the pointing device, the user is sensuously operating the pointing device while gazing at the GUI screen. This means that the user has to operate the device that is not in the field of view, and thus the operation cannot be intuitive and easy to understand.

Therefore, the following invention has been proposed as a configuration for inputting operation information in place of a widely known pointing device such as a mouse. First,
Japanese Unexamined Patent Publication No. 2000-154794 discloses an invention in which an operation input is performed by utilizing a blink of a person. Further, Japanese Patent Application Laid-Open No. 10-3150 discloses an invention in which a sound sensor detects a sound generated by tooth contact. However, all of these inventions are from the viewpoint of so-called wearable computing, and it is necessary to attach a special device to the human body, and the operation involves unfamiliar actions that a person does not usually do. That is, there is a problem that the system configuration becomes special and the operability is still difficult to be intuitively achieved.

Therefore, in terms of being able to perform intuitive operations without difficulty, it is preferable to use a display screen with a touch panel that allows direct pointing operations on the GUI screen as described above. . However, in an information processing device such as a personal computer, in order to realize a GUI operation using a touch panel, the user must own such a display with a touch panel, and the user must own the display. It will be troublesome. Also,
Since such a display device is expensive and involves an economic burden, it is disadvantageous in this respect as well.

Further, from the viewpoint of intuitive input operation, as shown in, for example, Japanese Unexamined Patent Publication No. 10-161801, the operation should be instructed by a voice uttered by a user. Can also be considered. In this case, as a system configuration, a general-purpose microphone that picks up a user's voice may be added. However, in this case, since the voice uttered by the user is used for the input operation, there is an inconvenience that only the contents related to the operation can be uttered during the operation.

[0007]

In view of the above, the present invention is directed to an information processing apparatus for the purpose of facilitating an intuitive input operation, even though the present invention is configured by a general-purpose apparatus as much as possible. It was decided to configure as follows. That is, display means for displaying and outputting an image on the display screen, image capturing means for capturing the captured image captured by the image capturing device, and image partial area of the display screen in the captured image captured by the image capturing means. And the point position recognition means for recognizing the position pointed by the pointer existing in the image partial area recognized by the image part recognition means as the point position on the actual display screen, and this point position recognition According to the point position recognized by the means, the information processing means capable of executing required information processing is provided.

Further, the information processing system is configured as follows. The information processing system according to the present invention comprises at least an imaging device and an information processing device, and the imaging device is provided at a position where the entire display screen of the information processing device can be imaged. Then, the information processing apparatus displays the image on the display screen by displaying the image, the image capturing unit that captures the captured image captured by the image capturing device, and the display in the captured image captured by the image capturing unit. Image part recognition means for recognizing the image part area of the screen, and point position recognition means for recognizing the position pointed by the pointer existing in the image part area recognized by the image part recognition means as the point position on the actual display screen. And an information processing means capable of executing required information processing according to the point position recognized by the point position recognition means.

Further, the program is configured as follows. That is, the image capturing procedure for capturing the captured image captured by the image capturing apparatus arranged to capture the display screen of the information processing apparatus, and the image of the display screen in the captured image captured by the image capturing procedure. An image partial recognition procedure for recognizing a partial area, a point position recognition procedure for recognizing a position pointed by a pointer existing in the image partial area recognized by the image partial recognition procedure as a point position on an actual display screen, and An information processing procedure capable of executing required information processing according to the point position recognized by the point position recognition procedure is configured to be executed by the information processing apparatus.

In each of the above structures, the image pickup device photographs the display screen of the information processing device. And
A graphical user interface image (GUI image) actually displayed on the display screen at the position indicated by a pointer (finger, pointer pen, etc.) existing in the captured image
Convert to the point position in. Then, required information processing is executed according to the point position. According to this, if the user operates the GUI screen by directly touching the display screen, information processing corresponding to this operation is executed. In short,
By providing the information processing apparatus with the imaging device, it is possible to perform a touch panel operation.

Further, the information processing apparatus of the present invention is configured as follows. That is, a display means for displaying and outputting an image on the display screen, a voice capturing means for capturing a voice signal picked up by the microphone device, and a voice signal captured by the voice capturing means for the display screen. Means for extracting the voice signal component of the contact sound generated by the contact of the physical pointer with the physical pointer, and an analyzing means for executing a required analysis process for the voice signal component of the contact sound extracted by the extracting means; An information processing means capable of executing required information processing based on the analysis result of the analysis means is provided.

As the information processing system of the present invention,
Also configure as follows. The information processing system according to the present invention includes a microphone device and an information processing device. The information processing apparatus includes a display means for displaying and outputting an image on a display screen, a voice capturing means for capturing a voice signal collected by the microphone device, and a voice signal captured by the voice capturing means. Extraction means for extracting an audio signal component of a contact sound generated when a physical pointer comes into contact with the display screen, and required analysis processing for the audio signal component of the contact sound extracted by the extraction means. The analysis means and the information processing means capable of executing required information processing based on the analysis result of the analysis means are provided.

Further, the program of the present invention is configured as follows. That is, it occurs when a physical pointer comes into contact with the display screen of the information processing device from the voice capturing procedure for capturing the voice signal collected by the microphone device and the voice signal captured by this voice capturing procedure. An extraction procedure for extracting the voice signal component of the contact sound, an analysis procedure for executing the required analysis processing for the voice signal component of the contact sound extracted by this extraction procedure, and a required procedure based on the analysis result of this analysis procedure. An information processing procedure capable of executing information processing is configured to be executed by the information processing apparatus.

In each of the above-mentioned configurations, the user directly touches the display screen of the information processing device with the pointer (finger, pointer pen, etc.). The contact sound is input by a microphone and analyzed. Then, information processing is executed according to the analysis result. This is G
This means that a touch panel-like operation of bringing the pointer into contact with the display screen on which the UI image is displayed is realized by using the sound generated when the pointer comes into contact with the display screen.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below. The following description will be given in the following order. 1. Example of system appearance 2. Configuration example of personal computer 3. Pseudo touch panel operation 3-1. Processing required for pointing operation based on captured image 3-1-1. Designation of display screen frame 3-1-2. Keystone correction of display screen frame 3-1-3. Coordinate conversion processing 3-2. Processing required for pseudo button operation based on audio signal 3-2-1. Speech signal recognition / analysis processing example 3-2-2. State transition according to analysis result 4. Processing operation when operating the pseudo touch panel 4-1. Software configuration example of image / voice recognition driver 4-2. 4. Processing operation example according to flowchart Modified example (pseudo touch panel operation by only pointing operation)

1. Example of System Appearance FIG. 1 is a perspective view showing an example of the appearance of the system according to the present embodiment. The personal computer 10 shown in FIG. 1 is an information processing apparatus as an embodiment of the present invention. In this figure, the personal computer
It is a so-called notebook type, and a liquid crystal display is adopted as a built-in display monitor. And the display screen 1 as this liquid crystal display
7a has a normal structure without a touch panel. However, in the present embodiment, as described below, the captured image captured by the camera unit 22 and the sound captured by the microphone 23 are used to be displayed on the display screen 17a. It is possible for the user to perform an input operation on the GUI image by directly touching a pointer such as a finger or a point pen.

To this end, the personal computer 10
For example, as shown in the figure, the external camera unit 22 is attached by the attaching tool 22-1. Then, the camera unit 22 is fixedly arranged as a captured image so that the entire display screen 17a is accommodated. The camera unit 22 has a structure capable of capturing a color image. Then, for example, a small one using an image pickup device such as a CCD (Charge Coupled Device) may be adopted. In addition, the camera unit 22 and the personal computer 10 are connected to a predetermined input terminal provided in the housing on the personal computer 10 side by, for example, a cable 22a,
The signal of the captured image obtained by the camera unit 22 is input to the personal computer 10. The input terminal in this case is, for example, a USB (Universal Serial Bus) terminal or an IEEE in addition to the analog video signal input terminal.
Various types of data interface terminals such as an E1394 terminal are conceivable and are not particularly limited.

Similarly, the microphone 23 is also
The cable 23a is connected to a terminal corresponding to a microphone input, such as an audio signal input terminal on the personal computer 10 side. As a result, the personal computer 10 can input the audio signal obtained by picking up the sound by the microphone 23. Here, in the present embodiment, the microphone 23 collects the sound generated when the user operates the pointer by directly touching the display screen 17a, for example, the sound when the display screen 17a is tapped or rubbed. There is a need to. Therefore, the microphone 23
Is installed at a position where the contact sound generated on the display screen 17a can be accurately collected. Further, considering that the contact sound generated on the display screen 17a should be more accurately collected, the microphone 2 having a required directivity is provided.
3 may be used.

If the personal computer 10 has a built-in microphone 23A from the beginning, the built-in microphone 23A may be used instead of the externally attached microphone 22. This also applies to the camera unit 22. That is, the personal computer 10
On the other hand, if the camera unit is attached in advance and the mounting structure that can be arranged so that the entire display screen 17a can be imaged is adopted, such a provided camera unit may be used. is there.

Further, in FIG. 1, the personal computer 10 which is the information processing apparatus of the present embodiment is
Although it is supposed to be a notebook type personal computer, it may be a so-called desktop type personal computer. Also, in the case of such a desktop type, the display monitor is not limited to the liquid crystal display, and may be a CRT.

Further, the present invention should not be limited to the configuration shown in FIG. 1 in terms of system configuration. For example, there is known a system in which a projector device and a personal computer are connected to each other and an image of the personal computer is displayed on a screen by the projector device. However, according to the present invention, a system of a personal computer including such a projector device is known. Can also be applied to. FIG. 15 shows a configuration example when the present invention is applied to a system for displaying an image on a computer by such a projector device. In this case, the personal computer 10 connects the video output terminal to the video input terminal of the projector device 70 so that the image signal from the personal computer 10 is input to the projector device 70. As a result, the projector device 70 enlarges and projects the same image as that displayed on the display screen 17a of the personal computer 10 onto the screen 71 for display.

Also in this case, the camera unit 22 and the microphone 2 are connected to the personal computer 10.
I am trying to connect with 3. In this case, the camera unit 2
2 is arranged so that the entire screen 71 fits within the captured image, and the captured image signal is input to the personal computer 10. Also, microphone 2
Reference numeral 3 is provided at a position where a "contact sound" generated by, for example, bringing the tip of a pointer rod 72 as a pointer, which will be described later, into contact with the screen 71 and performing some operation. Then, the collected audio signal is input to the personal computer 10. The pointer 72 is a pointer for performing an operation on the screen 71, and the user is asked to perform a GUI operation by directly contacting the tip of the pointer 72 with the screen 71. I am trying. Even in such a system configuration, as in the case of the system of FIG. 1 described below, the captured image of the screen 71 captured by the camera unit 22 and the sound of the contact sound captured by the microphone 23. By using the signal, screen 7
The GUI corresponding to the touch operation of the pointer on the image displayed on the screen 1 can be realized.

In the systems shown in FIGS. 1 and 15, the camera section 22 and the microphone 23 are connected to the personal computer 10 by a cable. It is conceivable that the configuration is such that the connection is made wirelessly by using such as. Further, particularly in the system of FIG. 15, it is conceivable that the input and output of the video signal between the personal computer 10 and the projector device 70 is performed wirelessly.

2. Configuration Example of Personal Computer FIG. 2 is a block diagram showing an internal configuration example of the personal computer 10 of the present embodiment.
The CPU 11 executes various processes in accordance with a program stored in the ROM 12 as firmware, an OS (Operating System) stored in the HDD 18, an application program, and the like expanded in the RAM 13. Further, the RAM 13 also appropriately stores data and the like necessary for the CPU 11 to execute various processes. These are CPU11, ROM12,
The RAM 13 is interconnected by an internal bus 25 together with a network interface 20, a data interface 21, an input / output interface 14 and the like which will be described later. The internal bus 25 is, for example, a PCI (Periphera).
l Component Interconnect) or a local bus.

The input / output interface 14 is provided for exchanging information between the internal bus 25 and devices connected to the input / output interface 14. In this case, the keyboard 1 is used.
5, mouse 16, display monitor 17, HDD1
8, the media driver 19, the camera unit 22, the microphone 23 (23A), etc. can be connected.

The operation signals supplied from the keyboard 15 and the mouse 16 are sent to the CP from the input / output interface 14.
It is designed to output to U11. In the CPU 11,
For example, under the OS program, these keyboards 15
Also, a required process corresponding to the operation signal from the mouse 16 is executed. In the present embodiment, the camera unit 22
The captured image signal output from the microphone 23
The GUI operation instead of the operation of the mouse 16 can be performed based on the information of the audio signal that is collected and output at (23A). When the personal computer 10 of the present embodiment is a notebook type, a touch pad and a click button are widely provided in this notebook type as operators in place of the mouse. Therefore, although not shown in FIG. 2, these touchpads and click buttons are used by the mouse 1
6 may be connected to the input / output interface 14.

A video signal for display is output to the display monitor 17 via the input / output interface 14, whereby the display monitor 17 is provided.
An image will be displayed on the display screen 17a.

Further, the input / output interface 14 includes
A hard disk drive (HDD) 18 having a hard disk as a storage medium is connected. CPU11
Is capable of recording or reading data, programs, etc. on the hard disk of the HDD 18. Then, the write data and the read data are transferred to the HDD 18 via the input / output interface 14.
And the internal bus 25 side. Particularly in this embodiment, the image / voice recognition driver 18a is installed and stored in the HDD 18. As will be described later, the image / speech recognition driver 18a performs a contact operation performed by the user on the display screen 17a of the display monitor 17 based on the image pickup signal from the camera unit 22 and the sound signal from the microphone 23. It is an application program that realizes processing for handling as input operation information.

Further, the media driver 19 is, for example, in the present situation, a driver corresponding to a specific type of media such as a CD-ROM and a DVD, and reads / writes data from / to the corresponding media. Run. The CPU 11 also executes the control for this purpose, and the write data and the read data are transferred to the HDD 18 and the internal bus 25 via the input / output interface 14.
Transmitted to and from the side.

The camera unit 22 in this case is provided as described with reference to FIG. 1,
For example, the image pickup device includes a CCD as an image pickup element. The captured image signal, which is the signal of the image captured by the camera unit 22, is input / output interface 1
4 will be input. Microphone 23
(Or the built-in microphone 23A) is also provided for the personal computer 10 as shown in FIG. The voice signal of the voice picked up by the microphone 23 is input as a digital audio signal through the input / output interface.

The net interface 20 is an interface for performing communication via a predetermined network, and for example, if it corresponds to the connection to the Internet using a telephone line, the hardware includes a modem or the like. In addition, LAN (Local Area Network)
Eth if it is via a network such as k)
It becomes an interface such as ernet.

The data interface 21 is, for example, S
CSI, USB, IEEE1394, etc.
It is used as an interface for communication with external peripheral devices by cable connection. The camera unit 22 and the like described above may be connected to the data interface 21.

By the way, the image / voice recognition driver 18 to be installed in the HDD 18 as described above.
“A” is an application program in which method procedures for realizing the functions described below are stored as a program. Then, the image / voice recognition driver 18
The application program as a includes a floppy (registered trademark) disk and a CD-ROM (Compact Disc Replay).
ad Only Memory), MO (Magnet Optical) disc, D
It can be temporarily (or permanently) stored (recorded) in a removable recording medium such as a VD (Digital Versatile Disc), a magnetic disk, a semiconductor memory.
Such a removable recording medium can be provided as so-called package software. Then, such a removable recording medium is loaded into the media driver 19 of the personal computer 10, for example, and played back, so that the H
It will be installed for DD18. In addition to installing the application program using the removable recording medium as described above, the personal computer 10 can be installed from another information processing device.
To the personal computer 10 online via a network such as a LAN or the Internet, and the personal computer 10 receives the application program thus transferred. , HDD 18 can also be installed.

3. Pseudo touch panel operation 3-1. Processing required for pointing operation based on captured image 3-1-1. Designation of Display Screen Frame As is well known, as a GUI operation, it is general to operate a pointing device (operating device) such as a mouse (and touch pad, click button, etc.). On the other hand, in the present embodiment, the GUI operation is realized by the pointing operation performed by directly touching the display screen instead of the operation by the pointing device. That is, a touch panel operation is realized. Moreover, in the present embodiment, even if the display screen does not have the structure of a touch panel, the microphone 23 (23A) produces a captured image of the camera unit 22 that captures the display screen 17a and a sound generated by various touch operations on the display panel. The touch panel-like operation described above is realized by executing a recognition process as will be described later with respect to the voice obtained by collecting the sound. Note that the touch panel-like operation as in the present embodiment will also be referred to as “pseudo touch panel operation” hereinafter.

In the "pseudo touch panel operation" of the present embodiment, the operation input based on the recognition of the picked-up image of the display screen 17a has a function of moving the cursor with the mouse, that is, "pointing operation". Have. The operation input based on the recognition of the contact sound obtained on the display screen 17a has a function as a button operation of a mouse. That is, "pseudo button operation" is realized. In the description of the present embodiment, as the "pseudo button operation", the left mouse button operation (DOS /
V system).

Therefore, first, the "pointing operation" will be described as the "pseudo touch panel operation" of the present embodiment. In order to realize the pointing operation (cursor movement operation) as the “pseudo touch panel operation” of the present embodiment, the following processing is required. That is, the display area of the display screen 17a in the captured image obtained by capturing the display screen 17a is recognized, which image area in the captured image is the pointer, and the pointer position pointed to by the recognized pointer is determined. Is recognized, and the coordinates of the recognized pointer position are converted into coordinate information on the GUI image actually displayed on the display screen. That is the process. When the coordinate information obtained by the processing is passed to the OS, the OS controls the cursor movement according to the coordinate information. The “pointer” in the above processing means the display screen 17
It is a physical object for actually touching and operating a, and specifically includes a user's finger or, for example, a rod-shaped pointer pen held and used by the user.

Then, as a processing, the "display screen 17a
In order to recognize the display area of the display screen 17a in the captured image obtained by capturing "," the initial setting described with reference to FIG. 3 is performed in advance.

FIG. 3 shows an initial setting procedure for recognizing the display area of the display screen 17a in the captured image. Note that this initial setting is performed under the program of the image / voice recognition driver 18a. The user sets the initial setting mode by, for example, a predetermined operation. Note that a pointing device such as a mouse may be used for this initial setting operation. In the initial setting mode, the captured image of the camera unit 22 arranged as shown in FIG. 1 is displayed on the display screen 17a of the personal computer 10. At this time, the user sees the captured image displayed on the display screen 17a and
As shown in (a), on the display screen 17a, the arrangement position and orientation are adjusted so that the entire display screen 17a of the personal computer 10 is displayed. That is, the personal computer 10 is displayed on the display screen 17a.
As a PC image portion 10-1 which is an image of
The display screen area 17a-1 which is the image of 7a is displayed.

When the image state as shown in FIG. 3A is obtained as described above, the user performs a predetermined operation for designating the image corner. Then, as shown in FIG. 3B, the image corner designating cursor CR1 is displayed on the display screen 17a. The user performs, for example, a drag operation on the view-corner designating cursor CR1 so that the apex portion of the view-corner designating cursor CR1 is displayed on the display screen area 17a-
It is positioned at the upper left corner of 1, and a determination operation is performed, for example.

Then, as shown in FIG. 3 (c),
Since the next image corner specifying cursor CR2 is displayed, the user similarly drags this image corner specifying cursor CR2 to position the apex portion at the upper right corner of the display screen area 17a-1 to perform the determining operation. To do. Furthermore, as shown in FIG.
Next image corner designation cursor C displayed as shown in (d)
Drag R3 to display its apex on the display screen area 17a.
The determination operation is carried out by locating it at the lower right corner of -1. Then, subsequently, as shown in FIG. 3E, the last image-corner designating cursor CR4 is displayed. Therefore, this image-corner designating cursor CR4 is dragged to display its vertex at the display screen area 17
The determination operation is performed by locating the lower left corner of a-1.

By thus designating the four corners of the display screen area 17a-1 with the respective image-corner designating cursors CR1 to CR4, as shown in FIG. That is, the coordinate positions of the vertices a, b, c, d of the display screen area 17a-1 in FIG. This means that the shape of the display screen area 17a-1 has been recognized as rectangular "frame information".

3-1-2. Keystone correction of display screen frame By the way, due to the arrangement position of the camera unit 22, the display screen 17a is not necessarily photographed from the front. Therefore, as the quadrangular shape (frame information) of the display screen area 17a-1 recognized as shown in FIG.
In most cases, it is not a regular shape such as a rectangle or trapezoid, but a more distorted shape. In the present embodiment, by executing the coordinate conversion process of converting the coordinates on the display screen area 17a-1 (frame information) into the coordinates on the actual display screen 17a, the actual pointer of the display screen 17a is changed. Although the point position is specified, if the coordinate transformation process is executed as it is by using the quadrangular shape (frame information) of the display screen area 17a-1 as described above, the calculation process therefor becomes complicated. On the contrary, accuracy will be lost. Therefore, in the present embodiment, correction processing is performed on the quadrangular shape (frame information) of the display screen area 17a-1 as described below.

FIG. 4 schematically shows the above-mentioned correction processing. FIG. 4A shows the frame information recognized by the image corner specifying operation described with reference to FIG. That is, the display screen area 17 similar to that shown in FIG.
A quadrangular shape as a-1 is shown. In this case, it is assumed that the frame information shown in FIG. 4A has a distorted quadrangular shape for the above-mentioned reason. For example, in the frame information shown in FIG. The side ab and the side cd opposite thereto are not parallel to each other. Also, in the same way,
It is assumed that the side ad in the vertical direction and the side bc facing the side ad are not parallel to each other.

In this embodiment, the side a of the frame information is
It is determined that b is the upper base and side cd is the lower base, and then these two sides are corrected so that they are parallel to each other. That is, the frame information is corrected by making it trapezoidal. As a result, the frame information (quadrangle abc shown in FIG.
d) is, for example, a quadrangle ABCD indicated by a broken line in FIG.
The shape is changed so as to be a trapezoid as shown by. As a specific example, let each vertex of the frame information before correction be a (xa, ya) b (xb, yb) c (xc, yc) d (xd, yd), and set the corrected frame information that is the frame information after correction. Let each vertex be A (XA, YA) B (XB, YB) C (XC, YC) D (XD, YD), YA ＝ YB ＝ (ya + yb) / 2 YC ＝ YD ＝ (yc + yd) / Shape correction is performed so that the relationship of 2 holds. However, as a condition, the relationship of XA = xa XB = xb XC = xc XD = xd is satisfied so that no error will occur in the coordinate conversion result, so the shape of the correction frame information that has been trapezoidally corrected is , The side AD = side BC is not necessarily a so-called isosceles trapezoid. As described above with reference to FIGS. 3 and 4, by recognizing the frame information and performing the trapezoidal correction process on this frame information, the display area (frame) of the display screen 17a in the captured image is processed. (Information) has been recognized.

3-1-3. Coordinate conversion processing The frame information is recognized as described above, and when an operation is performed with a pointer (finger, point pen, etc.) in the subsequent normal mode, the coordinates of the point position pointed to by the pointer are recognized. Then, the coordinate conversion processing for converting the coordinates of the point position into the coordinates on the actual display screen 17a is performed. That is, the processing described above is executed. Then, here, the coordinate conversion processing corresponding to the processing will be described with reference to FIGS. In order to execute coordinate conversion processing as processing,
As described as the previous processing, after recognizing which image area in the captured image is the pointer, and further described as the processing, recognizing the point position of the pointer with respect to the frame information on the captured image. However, this will be described later. Here, it is assumed that the recognition processing of the point position of the pointer has already been properly performed.

Here, FIG. 5A shows the correction frame information obtained as described with reference to FIG. Then, it is assumed that coordinates (indicated by X in the figure) called input (xin, yin) are obtained by the user operating the pointer on the display screen 17a. This coordinate input (xin, yin) is
For example, the coordinates (in-image coordinates) as the image position detected in the effective display area of the captured image. Then, depending on the processing according to the present embodiment, this in-image coordinate inpu
By using t (xin, yin) as shown in FIG.
Actual coordinates outp on the display unit 17a where the UI image is displayed
It will be converted into ut (xout, yout), and the case of converting the y coordinate will be described first with reference to FIG. In this case, the y-coordinate (yout) of the real coordinates is the height height_img of the trapezoid as the correction frame information shown in FIG.
Vertical width of the display screen 17a shown in (b) height_d
It can be determined by the ratio with sp. That is, the y coordinate (yout) of the real coordinates can be calculated by the formula shown as yout = (yin-YA) * height_dsp / height_img (Equation 1).

Subsequently, the x coordinate is to be obtained. For this purpose, first, the correction frame information and the display screen 1 are displayed.
The position of 7a in the x-axis direction (horizontal direction) is aligned with a predetermined reference line. FIG. 6 schematically shows this reference position alignment. First, the display screen 17a shown in FIG. 6 (b) is shown as a rectangle composed of vertices W, X, Y and Z. A midline M0-M1 connecting the midpoints Z1 of ZY may be used.

On the other hand, although it is the reference line XN of the correction frame information shown in FIG. 6A, the trapezoid as the correction frame information is not necessarily the isosceles trapezoid as described above. Therefore, the point N0 on the side AB and the point N1 on the side CD of the correction frame information
As the reference XN to be formed by connecting with, the x-coordinates (XA, XB, XC, X) of the vertices A, B, C, D of the correction frame information are formed.
D), XN = (XA + XB + XC + XD) / 4 (Equation 2).

After the reference line XN of the correction frame information is obtained as described above, the x coordinate (xin) of the in-image coordinates is converted to the x coordinate (xout) of the actual coordinates. . FIG. 7 schematically shows this conversion of the x coordinate.

Here, in the trapezoid ABCD which is the correction frame information, the width at the height position passing the y coordinate (yin) is w.
Let idth_img be the x-coordinate (xout) of the real coordinate, and the width width_img of this trapezoid ABCD (correction frame information) and the display screen 1
It can be said that it is determined by the ratio with the width width_dsp of 7a in the horizontal direction. However, the trapezoid ABCD that is the correction frame information
The width width_img of is changed depending on the value of the y-coordinate (yin), so it is calculated as follows. That is, the x coordinate at the y coordinate (yin) for the point on the side DA is "x0", and the x coordinate at the y coordinate (yin) for the point on the side BC is "x1". x0 = (XD-XA) * (yin-YA) / height_img + XA x1 = (XB-XC) * (yin-YA) / height_img + XC width_img = x1-x0 ... (Expression 3) It is made to be calculated by a formula. And
If you use the value of width_img obtained in this way, for the x coordinate (xout) of the real coordinate, xout = (xin-XN) * width_dsp / width_img + width_dsp / 2 ... (Equation 4) Can be calculated. Thus, FIG.
~ If the arithmetic processing is performed as described with reference to Fig. 7, the in-image coordinates in the display screen area 17a-1 in the captured image
input (xin, yin) is the actual coordinate outp on the actual display 17a
Can be converted to ut (xout, yout). The coordinate conversion process described with reference to FIGS. 5 to 7 is merely an example, and may be a process using another arithmetic expression, for example.

3-2. Processing required for pseudo button operation based on audio signal 3-2-1. Example of Recognition / Analysis Processing of Audio Signal Next, the “pseudo button operation” will be described as the “pseudo touch panel operation” of the present embodiment. The "pseudo button operation" in this embodiment means
As described above, the operation information corresponding to the operation of the left mouse button is generated based on the contact sound generated when the user operates the pointer while directly touching the display screen 17a. Is. To this end, from the sounds picked up by the microphone 23,
It is necessary to perform the voice signal component recognition processing for recognizing the voice signal component as the "touch sound" by the pointer operation. By this voice signal component recognition processing, for example, a "contact sound" can be distinguished from other voice signal components such as the user's uttered voice. Then, after recognizing the voice signal component of the “contact sound” in this way, an analysis process of analyzing the generation state of the contact sound generated by the pointer operation by the user is performed.
That is, the generation state of the “contact sound” is recognized, and what operation is assumed to have been performed as the operation of the left mouse button is determined according to the recognition result.

First, the voice signal component recognition process as the above process is performed. In this embodiment, the frequency band characteristic of the voice signal component as the "contact sound" should be used for the analysis process in the initial setting. It shall be registered as a voice signal component. For example, the user
A registration mode for registering a voice signal component to be used in analysis processing is set. And under this registration mode,
The user at least makes a sound by hitting the display screen 17a or rubbing the display screen 17a. The sound thus generated is picked up by the microphone 23 and input to the personal computer 10. In the personal computer 10, the image / voice recognition driver 18
According to the program of a, the frequency band characteristic is detected,
The detected frequency band characteristic is registered as an audio signal component to be used in the analysis process. Then, when the contact operation by the pointer is actually performed on the display screen 17a, the contact sound generated by the contact operation is picked up by the microphone 23 and taken in as an audio signal. The audio signal is filtered by passing only the frequency band registered as described above. For this, only the audio signal component as the contact sound is extracted. That is, it means that the voice signal component recognition process is performed.

Then, the analysis processing as the above processing is executed as follows. This analysis processing is schematically shown in FIG. For example, in the personal computer 10, the voice signal component of the “contact sound” generated by the user's touch operation on the display screen 17 a is converted into the microphone 23.
The audio signal input from is separated and extracted. Then, when analyzing the audio signal component, two types of threshold values are set for the level of the audio signal component of the contact sound. That is, as shown in FIG. 8A, the first threshold value ± t based on the predetermined absolute value level with reference to the 0 level.
A threshold value h0 and a second threshold value ± th1 based on a predetermined absolute value level which is considered to be larger than the first threshold value ± th0 are set. In this case, as shown on the right side of FIG. 8A, the range of the absolute value level smaller than the first threshold value ± th0 is recognized as the “contact sound” being in the silent state.
Further, the first threshold value ± th0 or more, and the second threshold value ± th0
In the range of the absolute value level smaller than th1, the “contact sound” is recognized as a weak sound. Further, the absolute value level equal to or higher than the second threshold value ± th1 is recognized as strong sound.

The waveform shown in FIG. 8A is an audio signal component of a contact sound obtained by the user performing a contact operation on the display screen 17a. Personal computer 1
At 0, regarding the level of the audio signal component of this contact sound,
A comparison is made with the first threshold value ± th0 and the second threshold value th1. For example, as the waveform shown in FIG. 8A, in the section A, the first threshold value ± th0 and the second threshold value th are
During 1 the peak level is obtained. Therefore, in this case, as shown in FIG. 8B, it is determined that a weak sound is generated as the contact sound. Further, since the peak level equal to or higher than the second threshold th1 is obtained in the section B, it is determined that a strong sound is generated as the contact sound. Also in the section C, since the peak level is obtained between the first threshold value ± th0 and the second threshold value th1,
It will be determined that a weak sound has occurred. Further, between the section A and the section B, and between the section B and the section C, there is a 0 level or an absolute value level smaller than the first threshold value ± th0. It
In this way, as the analysis of the contact sound in the present embodiment, the level of the contact sound is determined.

Further, such a comparison between the contact sound level and the threshold value is supposed to be carried out at every predetermined time t, and in the section at each time t, the three-step judgment of silence, weak sound and strong sound is carried out. To be done. Therefore, according to FIG. 8B, a case where it is determined in the section A that the sound is weak with a time length of time t × 2 is shown. In section B,
The case where it is determined that the sound is strong over a time length of time t × 4 is shown. Further, in section C, time t × 9
That is, it is determined that the sound is weak over the length of time. In the present embodiment, such a time length of sound level determination is also treated as one of the analysis results.

3-2-2. The state transition according to the analysis result, and based on the analysis result of the contact sound obtained as described above with reference to FIGS. 8A and 8B, for example, as shown in FIG. Is taken. FIG. 9 shows the state transition of the mouse left button operation according to the generation state of the contact sound. For example, first, when a silent state is awaited for input of a contact sound, no action is taken in a non-responsive state as shown in step S1. Then, for example, if it is determined from the state of step S1 that a weak sound is generated, the state shifts to a state in which the left mouse button is pressed, as shown in step S2. Then, for example, if the generation of this weak sound ends within a predetermined time and changes to a silent state, as shown in step S3,
Transition to the state where the left button is released. That is, when a weak sound is generated within a predetermined period of a short time, as a result of executing the processing of steps S2 → S3,
It is treated as if the left mouse button was pressed and released in a short time. In other words, it is treated as if the one-click operation was obtained. In FIG. 8, for example, the analysis result of the section A corresponds to this. That is,
Since the weak sound is generated in a relatively short time t × 2, as shown in FIG.
The click operation is occurring.

On the other hand, in step S2, from the state in which the action is taken by pressing the left button of the mouse,
If the weak sound is said to have continued for a certain period of time as it is, it is treated as if the state in which the left button of the mouse is held is continued as shown in step S4.
Then, in the state corresponding to this step S4, if the user is moving the pointer while touching it on the display screen 17a, this is a drag operation. Then, if it is changed to silence under the state of step S4, as shown in step S3, the state transits to a state in which the left button is released, and if, for example, a drag operation has been performed so far, this The drag operation is canceled. In the case of FIG. 8, such a drag operation corresponds to the section C.
In this section C, the weak sound is continued for a relatively long time t × 9, and therefore, the drag operation occurs as shown in FIG. 8C.

If it is determined in step S2 that the left button has been pressed and the weak sound has changed to a strong sound, the process proceeds to step S5 and is treated as if a double click had been performed. become. After the state of step S5, even if strong sound continues or changes to weak sound, as shown in step S6, a transition to a non-responsive state is made and a double-click operation is performed reliably. I am careful to be seen. Also after this,
When a silent state is reached, a transition is made to a non-responsive state as step S1.

If it is determined that the sound has changed to the strong sound under the non-responsive state corresponding to the silence in step S1, the process proceeds to step S7, and it is considered that the double click has been performed. . Also in this case, even if the strong sound continues or the sound changes to a weak sound thereafter, as shown in step S8, the state is changed to the non-responsive state. Further, when the state of step S8 is returned to the silent state, the state is changed to the non-responsive state as step S1. In FIG. 8, the section B corresponds to the double-click operation in step S5 or step S7. In section B, a strong sound is generated over a period of time t × 4, but a double-click operation is generated corresponding to the first period of time t × 2, for example. Then, the non-reaction state is set in correspondence with the remaining time t × 2.

In the present embodiment, for example, the “pseudo button operation” is performed by analyzing the contact sound described with reference to FIGS. 8 and 9 and executing the state transition as the left button operation of the mouse based on the analysis result. Specifically, it is assumed that the contact operation is specifically performed in the following manner. Considering what kind of operation a touch sound is generated as a touch operation on the display screen 17a using the user's pointer, it is generally that an operation of tapping on the display screen 17a and a display screen It can be divided into an operation of rubbing and moving on 17a. When the display screen 17a is tapped, a tapping sound having a loudness corresponding to the tapping strength is generated, and when the display screen 17a is rubbed with the pointer, the display screen 17a must be touched by the pointer. Therefore, a certain amount of sliding noise is generated.

Therefore, first, regarding the click operation,
The user is asked to tap the display screen 17a. As a result, a short weak sound is generated as the contact sound, and for example, step S in FIG.
The click operation can be generated as described by the flow of 2 → S3. In the drag operation, the pointer is brought into contact by tapping the drag start position on the display screen 17a as described above, and the pointer is moved in any desired direction while keeping the contact state. I will. By doing so, first, a light sound as a tapping sound is generated by tapping the drag start position, and then a weak sound is generated by rubbing on the display screen 17a. Therefore, step S2 →
A state transition as a drag operation can be obtained by the flow of S4. Then, in the double-click operation, the display screen 17a is strongly tapped. in this case,
Since a strong sound is generated as the contact sound, the state transition to the double-click operation described in step S7 or step S5 of FIG. 9 can be obtained.

Assuming that the left button click operation is simulated in accordance with the operation to be performed by the user as described above, the silence / sound shown in FIG.
It is preferable that the first threshold value ± th0 and the second threshold value th1 for determining weak sound / strong sound can be variably set by the user. That is, the display screen 17
The volume of the contact sound generated when hitting or rubbing a is different for each user, and the sensitivity of the microphone 23 is also different depending on the device actually used as the microphone. In addition, the display screen 17
The volume of the contact sound varies depending on the physical structure such as the material of the surface as a. Therefore, according to such a difference in the conditions, the first threshold value ± th0 and the second threshold value are set so that the state in which the user can easily operate is obtained.
If the threshold value ± th1 is set, a more accurate reaction to the operation can be obtained.

4. Processing operation when operating the pseudo touch panel 4-1. Example of Software Configuration of Image / Voice Recognition Driver Next, based on the above description, the processing operation corresponding to the actual pseudo touch panel operation (touch operation on the display screen 17a by the pointer) will be described. . Therefore, first, image / image, which is a program for realizing the pseudo touch panel operation of the present embodiment,
FIG. 1 shows the program configuration of the voice recognition driver 18a.
It will be described with reference to 0.

FIG. 10 shows the image / voice recognition driver 18a.
3 schematically shows the program configuration of. As the image / speech recognition driver 18a, as shown in FIG.
0 and a mouse interface application 60.

The image / voice recognition module 30 is composed of a voice recognition module section 40 and an image recognition module section 50. Speech recognition module section 4
Reference numeral 0 is a program portion for taking in a voice signal input from the microphone 23 (23A) and executing the analysis process for the contact sound shown in FIG. 8, for example.
For this purpose, an audio capture module 41 and an audio analysis module 42 are provided.

The audio capture module 41 converts the analog voice signal input from the microphone 23 (23A) into audio data of a required format suitable for analysis processing, and passes it to the audio analysis module 42. Depending on the audio analysis module 42, the analysis processing described with reference to FIG. 8 is executed. That is, the filter process for extracting the signal component of the frequency characteristic of the contact sound registered in advance from the audio data received from the audio capture module 41 is executed. Subsequently, the level of the frequency signal component of the contact sound extracted by this filter processing is compared with a preset first threshold value ± th0 and a second threshold value ± th1 at intervals of time t. Then, the judgment is made in three stages of silence, weak sound, and strong sound. Then, the determination result for silence, weak sound, and strong sound thus determined at each time t is passed to the button operation generation module 62 of the mouse interface application 60.

Further, the image recognition module section 50 has a function of performing recognition processing on an image signal as a captured image input from the camera section 22 and acquiring coordinate information of a pointer. It is configured by a program as the module 51 and the coordinate conversion module 52. Video capture module 51
Captures a video signal of a captured image input from the camera unit 22 and converts it into frame-based video data in a required format. Then, this video data is transferred to RAM1.
Writing is performed for each frame in a predetermined area of No. 3. Then, depending on the coordinate conversion module 52, the frame information shown in FIG. 3 is recognized from the video data written in the RAM 13 frame by frame as described above, and further the trapezoidal correction of the frame information shown in FIG. 4 is performed. To be processed. Then, first, the tip position of the pointer is recognized as described later. The tip position of the pointer is the tip position of the finger or the pointer pen as the pointer, and actually corresponds to the position on the display screen 17a pointed by the pointer. The recognition processing of the pointer tip position will be described later. Then, in this way, the position of the tip of the pointer on the captured image (coordinates in the image: see FIG. 5A)
Is recognized, predetermined calculation processing is performed as described with reference to FIGS. 5 to 7 so as to correspond to the coordinates (actual coordinates: see FIG. 5B) on the actual display screen 17a. Perform coordinate conversion processing. The coordinate information obtained by this coordinate conversion process is used as the mouse interface application 6
It is passed to the cursor position information generation module 63 at 0.

In this way, the image / speech recognition module 30 is operated by the mouse interface application 6
For 0, the analysis result of the audio signal from the voice recognition module 40 and the coordinate information obtained by the coordinate conversion processing from the image recognition module unit 50 are transferred. Then, depending on the button operation information generation module 62 in the mouse interface application 60, based on the analysis result received from the image / speech recognition module 30, there is no sound / weak sound / sound regarding contact sound.
The state transition of the generation of strong sound is recognized, and in accordance with the recognition result, as shown in FIG. 9, it is converted into operation information for pressing / releasing the left mouse button, and the operation information transmitting module 61 receives hand over. In addition, depending on the cursor position information generation module 63 in the mouse interface application 60, a process for setting the coordinate information received from the image recognition module unit 50 as the coordinates of the current cursor position is executed. The information is passed to the operation information transmission module 61.

Depending on the operation information transmission module 61,
The operation information for pressing / releasing the left button of the mouse and the coordinates of the cursor position, which have been transferred as described above, are converted into operation information that can be processed by the OS and transferred to the OS. The OS receives this operation information, so that the GU
The cursor movement display on I, the display change according to the click and double-click operations, and the required information processing are executed. For example, switching of the active window on the GUI in response to a click, opening of a file in response to a double-click, activation of an application, etc. are performed.

4-2. Processing Operation Example According to Flowchart Next, as processing operation based on the image / voice recognition driver 18a having the program structure shown in FIG.
The processing corresponding to the user performing the pseudo touch panel operation on the display screen 17a will be described with reference to FIGS.

FIG. 11 is a process to be executed by the image recognition module 50, the voice recognition module 40, and the mouse interface application 60 described with reference to FIG. Are shown in parallel. For confirmation, the CPU 11 executes the image / voice recognition driver 18 in the process shown in FIG.
It is executed according to the program as a.

First, the processing according to the image recognition module 50 will be described. Here, first, as the process of the video capture module 51, as shown in step S101, a capture process for a video signal is executed. That is, the video signal of the captured image input from the camera unit 22 is converted into image data in a predetermined format that is compatible with the subsequent pointer detection and coordinate conversion processing, and is sequentially frame-by-frame with respect to a predetermined area of the RAM 13, for example. I will write.

The subsequent processing in step S102 and thereafter is processing according to the coordinate conversion module 52. Step S1
In 02, the pointer is detected from the video signal of the captured image in the frame unit written in the RAM 13 as described above. It should be noted that the pointer here means that the user can display on the display screen 17 as can be understood from the above description.
It is a physical entity that should be used to perform a pseudo touch panel operation on a, and is, for example, the user's finger or a thin rod-shaped pointer pen. In addition, the detection of the pointer here refers to finally detecting the position as a point pointed by the pointer on the captured image, that is, the position (coordinates) of the tip of the pointer. Then, the process of step S102 is executed as shown in the flowchart of FIG. 12, for example.

Before describing the processing of FIG. 12, the initial setting regarding the pointer will be described. In order for the pointer detection process shown in FIG. 12 to properly detect the pointer, it is necessary to know the color of the pointer in advance. Therefore, prior to performing the pseudo touch panel operation as the present embodiment, the color information of the pointer is registered at the time of initial setting. For this purpose, for example, the user sets the pointer color information registration mode by a predetermined operation. In this registration mode, the image captured by the camera unit 22 is displayed on the display screen 17a of the personal computer 10. Then, under this state, for example, if the user uses his / her own finger as the pointer, the user picks up his / her finger with the camera unit 22 and displays it on the display screen 17a. If an object such as a pointer pen is used as the pointer, the pointer pen is displayed on the display screen 17a.

Then, the user positions the cursor for color acquisition in the image area of the pointer which is imaged by the camera section 22 and displayed on the display screen 17a as described above, and performs the registration operation. As a result, the color information of the captured image area at the position pointed to by the color acquisition cursor is registered as pointer color information. For example, if the pointer is the user's finger, the color information as the skin color for this finger will be registered.
If the pointer is a pointer pen and the color is blue, the blue color is registered as color information.

The pointer detection process shown in FIG. 12 is performed as follows. First, in step S401, the captured image data in frame units written in the RAM 13 is divided into a plurality of preset color regions of a plurality of types. The "plurality of color areas" here means
For example, as shown in FIG. 13, it refers to a plurality of areas obtained by dividing a color expressed in a YUV (luminance / color difference) chromaticity space into a plurality of predetermined areas.

In the following step S402, each color area divided in step S401 is compared with the registered color area corresponding to the pointer color information registered as described above. In the registered color area, for example, when the pointer is the user's finger and the pointer color information registered as described above is the color information as a certain skin color, it is indicated by the diagonal lines in FIG. , + Y, -U corresponding to the pointer color information as the skin color,
An area within a certain range in the −V quadrant is set as the registered color area R. The registration color area R may be set, for example, when the pointer color information is registered.

Then, in the next step S403,
As a result of the comparison in step S402, the registered color area R in the frame image area as the captured image is displayed.
Is present. Here, when a negative result is obtained assuming that the registered color region R does not exist,
Although the routine exits from step S102 as it is, if a positive result is obtained because the set color region R exists, the process proceeds to step S404.

Here, as a case where a positive result is obtained in step S403,
It may be determined that the registered color region R exists.
In such a case, one of the plurality of registered color areas R can be considered to be the registered color area R actually obtained by imaging the pointer. Then, in the present embodiment, a plurality of determined registered color regions R
Among them, the registered color area having the largest area is specified as the registered color area R obtained by actually capturing an image of the pointer. The process for this is the process of step S404.

That is, in step S404, the registered color region R having the largest area is detected from the plurality of registered color regions R determined to exist in step S403. Then, the detected registered color area R is set as an image partial area (pointer area) of the pointer in the captured image. When recognizing the pointer area, another recognition method may be used, or the above step S
Other recognition methods, including the recognition method as 404, may be used together. For example, since the pointer moves on the display screen 17a as needed, it is possible to incorporate a motion detection process as needed.

In the next step S405, the coordinate range in the picked-up image data for the pointer area detected as described above is acquired. Here, FIG.
In the figure, the captured image that is the processing target is shown. In addition, in this figure, a state in which the display screen area 17a is displayed as the PC image portion 10-1 of the personal computer 10 in the captured image is shown. Image data in such an image state is written in the RAM 13 in units of frames, and is to be processed. Then, for example, when the user uses his / her finger as the pointer, the above step S404.
Depending on the processes up to, the image area portion as the image GH of the user's hand is obtained as the coordinate range of the pointer area. Here, when the user is performing a touch operation on the display screen 17a with the finger as a pointer, the point position actually pointed is, for example, the tip of the index finger of the user. Therefore, in order to specify a point (coordinate) as a point position from the coordinate range of the pointer area on the captured image, it is sufficient to detect the position (coordinate) of the tip Pt as the fingertip. Step S406 following step S405 is a process for this purpose.

Here, for example, considering an actual pointer operation using the user's finger as an example, the user points his or her index finger upward as can be seen from the image GH of the user's hand in FIG. 14, for example. The pointing operation will be performed with. It should be noted that the same applies when a pointer pen or the like is used, for example, the operation is performed with the tip of the pointer facing upward. Therefore, in obtaining the position (coordinates) of the tip Pt in step S406,
The following processing will be executed. Depending on the information of the coordinate range acquired in step S405,
For example, the shape of the pointer in the captured image can be specified. Then, from the coordinate value group forming the specified shape, for example, the coordinate value existing on the uppermost screen side in the captured image data may be acquired as the coordinate of the tip portion Pt. Alternatively, from the shape specified by the coordinate range of the pointer, further specify the coordinate range of the image portion extending in the longest rod shape, and select the rod-shaped tip from the coordinate value group forming the rod-shaped image portion. It is also possible to acquire the coordinates of the position as the coordinates of the tip Pt.

The description returns to FIG. After the pointer detection processing in step S102 shown in FIG. 12 is executed, it is determined in step S103 whether the pointer is properly detected. That is, step S102
It is detected whether or not the coordinates of the pointer area and the tip portion Pt are detected by the processing of. Here, for example, as a result of executing the processing shown in FIG.
If the processing after 404 is not executed, a negative result is obtained because the pointer is not detected. In this case, the process routine shown in this figure is once exited and the process returns to step S101 again. On the other hand, as the processing shown in FIG. 12, the processing after step S404 is executed, and finally the coordinate leading end portion P
If the coordinate of t has been obtained, a positive result is obtained in step S103, and the process proceeds to step S104.

In step S104, as shown in FIG.
In step S406 in step S406
A process of converting the coordinates of t (coordinates in the image) into the coordinates (actual coordinates) on the display screen 17a is executed. The process for this is as described with reference to FIGS. Then, in the next step S105, in response to the acquisition of the actual coordinates in step S104,
Generate an event for the mouse interface application 60.

As the processing according to the image recognition module 50, for example, the processing of steps S101 to S105 is repeatedly executed for each frame cycle. This causes an event to be thrown to the mouse interface application 60 as long as it is detected that the pointer is present in the captured image, for example. Then, for example, if the pointer moves, an event will occur each time the actual coordinates to be converted are acquired according to this movement.

Next, processing according to the voice recognition module 40 in FIG. 11 will be described. This process
This is shown as the processing of steps S201 to S203 on the left side in FIG. First, in step S201, as the processing of the video capture module 41,
Executes capture processing for audio signals. That is, for example, regarding the audio signal input by being picked up by the microphone 23 (23A),
If necessary, conversion processing to a digital signal is performed, and then conversion is performed to required audio data suitable for subsequent analysis processing. Then, for example, writing is sequentially performed to the work area secured in the RAM 11.

Then, the processes following step S202 are processes according to the audio analysis module 42. In step S202, the analysis process is executed as described with reference to FIG. That is, first, the input audio data is filtered to separate and extract only the signal component corresponding to the preset frequency band of the contact sound. Then, the level of the signal component of the extracted contact sound is compared with preset threshold values (first threshold value ± th0 and second threshold value ± th1). Further, as the comparison result, an analysis result in three stages of silence, weak sound, and strong sound is obtained.
It should be noted that such an analysis process is performed at a timing of every predetermined time t as described with reference to FIG. Then, in the next step S203, the above step S20
At 2, the mouse interface application 60 is caused to generate an event indicating that the analysis result information (silent sound, weak sound, strong sound) has been obtained.

Next, the processing operation according to the mouse interface application 60 will be described. This process is performed in steps S301 to S in the center of FIG.
This is shown as the processing of 307. First, step S3
The processing of 01 to S302 is processing according to the cursor position information generation module 63. In step S301, step S1 is performed according to the image recognition module 50.
It is waiting to receive the event that occurred at 05. Here, until the event from the image recognition module 50 is received, the process of steps S302 → S302 is skipped and the process proceeds to the process of step S303. If obtained, step S3
Go to 02.

In step S302, a process for fetching the information of the actual coordinates of the pointer which is assumed to be acquired in the previous step S104 is executed. Then, in the subsequent step S303, cursor position information is generated in correspondence with the information of the actual coordinates of the pointer that has been taken in. After this, the process proceeds to step S304.

Steps S304 to S306 are processes according to the button operation information generation module 62. In step S304, according to the process of the voice recognition module 40, it waits for the reception of the event that has occurred in step S203. If it is determined in step S304 that an event has not been received, the process of steps S305 → S306 is skipped and the process proceeds to step S307. On the other hand, if a positive result is obtained because the event is received in step S304, step S30
Go to 5.

In step S305, the analysis result (silent sound, weak sound, strong sound) obtained in the previous step S202 is fetched. Then, depending on the processing of the next step S306, the above step S202
After recognizing the state transitions of silence, weak sound, and strong sound generation based on the analysis result captured by, operation information for pressing / releasing the left button of the mouse shown in FIG. 9 is generated. Then, the process proceeds to step S307.

The processing of step S307 is processing according to the operation information transmission module 61. That is, the process of step S307 is the same as steps S303 and S30 described above.
The current cursor position information and the left button operation information, which is supposed to be generated by 6, are converted into data of a structure that can be processed by the OS, and the processing for passing to the OS is executed. Depending on the passed information, the GU
The various processes corresponding to the I operation are executed as described in the description of FIG. That is, the processing corresponding to the operation corresponding to the movement of the mouse and the operation corresponding to the pressing / release of the left mouse button is executed.

By implementing the pseudo touch panel operation of the present embodiment as described above, for example,
Even if a display device with a touch panel is not prepared, a touch panel-like operation of directly touching the display screen for operation can be easily realized. Touch panel operation is different from, for example, mouse operation or operation based on the idea of so-called wearable computing,
Since this is an intuitive operation, it is possible to perform a more convenient GUI operation.

In order to realize such a pseudo touch panel operation in the case of the present embodiment, the user only needs a general-purpose personal computer in which the image / voice recognition driver 18a is installed, a camera device and a microphone. Will be good. As the camera device, for example, a CCD camera for a personal computer which has been widely used so far may be used, and a general microphone which has been conventionally used may be used as a microphone. Further, especially for a microphone, if it is built in a personal computer, even if it is used, it can be sufficiently put into practical use. For this reason, the user can prepare the necessary devices without paying much attention to the financial burden. For example, a display device with a touch panel is very expensive under the present circumstances, but in the case of the present embodiment, it is possible to obtain a system capable of performing a touch panel operation more easily.

Further, in the present embodiment, the "contact sound" on the display screen 17a is recognized as the voice recognition. For example, a configuration is known in which GUI control is executed in response to a voice uttered by a user. In this case, however, the user cannot inadvertently speak, so that, for example, a conversation is sufficient during operation. However, in the case of the present embodiment, since the GUI control is performed based on “contact sound” having a completely different frequency band characteristic, etc., from the uttered voice, the above-mentioned problem occurs. Will be resolved.

5. Modified Example (Pseudo Touch Panel Operation by Only Pointing Operation) In the above-described pseudo touch panel operation according to the present embodiment, the pointing operation by the user's pointer is made to correspond to the movement operation of the mouse, and the touch operation corresponding to this pointing operation is made. The sound information was made to correspond to the left mouse button operation as "pseudo button operation". However, in the present embodiment, it is possible to realize, for example, a “pseudo button operation” such as clicking without using the above-mentioned contact sound information. Hereinafter, this point will be described as a modified example.

An example of the operation mode in this case will be described first. For example, even in this case, when the user performs a touch operation on the display screen 17a using the pointer, the coordinates of the tip Pt of the pointer in contact with the display screen 17a are finally determined by the cursor position information. Is treated in the same manner as in the above embodiment. Therefore, as the pointer touching the display screen 17a moves, the position of the tip Pt of the pointer (see FIG.
The cursor displayed on the display screen 17a is moved in accordance with (see). Then, in this case, when the user wants to perform the click operation, first, the tip portion Pt of the pointer is stopped at the point position where the user wants to click, for a certain period of time, and then,
For example, the display screen 17a is once removed. Then, after this, the tip portion Pt of the pointer is returned to the same point position to be clicked. That is, the player operates the same position on the display screen 17a once at a certain speed.

Such an operation is performed on the picked-up image at the tip portion Pt of the pointer at the coordinates of a certain point position.
Is temporarily stopped, and thereafter, the tip portion Pt of the pointer appears to be located at the coordinates of the position where it was temporarily stopped. Therefore, for example, depending on the cooperation with the image recognition module unit 50 and the mouse interface application 60 shown in FIG. 10, the tip Pt of the pointer in the captured image data for each frame is detected.
Is detected. Then, as a result of the motion detection, the movement is stopped at a coordinate A for a certain time or longer, and thereafter the tip P is moved away from the coordinate A.
When it is determined that the state transition in which t moves → the tip Pt moves to the coordinate A again within a certain time from the above state, the coordinate A is clicked as a click point. The operation information is handed over to the OS. Regarding double-click,
For example, after the tip portion Pt of the pointer is stopped at the point position where the user wants to click for a certain time or more, the point position is hit twice in succession. Then, when the condition of the motion detection result corresponding to this is obtained, it may be handled as if the double click was performed.

The various processes for implementing the pseudo touch panel operation according to the present invention are not limited to those described above, and can be changed as appropriate. For example, the operation procedure for setting the frame information shown in FIG.
The calculation method and the like for the coordinate conversion processing described with reference to FIG. 7 are also conceivable. In addition, the content of the analysis process for the contact sound shown in FIG. 8 and the state transition of the left mouse button depending on the analysis result shown in FIG. I don't mind. For example, in the analysis process shown in FIG.
The contact sound level is classified into three levels of silence, weak sound, and strong sound. For example, depending on the operation mode,
It is also conceivable that there are two stages or four or more stages. Furthermore, it is conceivable that the program configuration and the processing operation for realizing the pseudo touch panel operation shown in FIGS.

[0100]

As described above, the present invention recognizes the point position, which is the position on the display screen pointed by the pointer, from the captured image obtained by capturing the display screen, and recognizes the recognized point. It is configured to execute required information processing in response to the GUI operation according to the position. With such a configuration, even if the display screen does not have a structure as a touch panel, it is possible to perform an operation similar to that of the touch panel. Generally, if the display screen does not have a structure as a touch panel,
Although an operator such as a mouse or a trackpad is operated, such an operation is not necessarily an intuitive operation because the operator is not in the field of view of the user. However, according to the present invention, even if the display screen does not have a touch panel, intuitive touch panel operation is possible. Further, as a hardware configuration for realizing such a touch panel operation of the present invention, a camera device (image pickup device) may be provided for a general-purpose information processing device such as a personal computer. In other words, since it is expensive and does not require a special device, the user can easily set up the system. Further, since a computer display with a touch panel and the like are considerably expensive, the system according to the present invention can be easily assembled even when compared with the purchase of such a computer display.

Further, according to the present invention, the contact sound generated by the physical pointer touching the display screen is picked up by the microphone and analyzed, and the GUI operation is responded to based on the result of the analysis. The required required information processing is executed. In this case, intuitive operations such as rubbing the display screen to move the cursor and moving the pointer, or tapping the display screen to click / double-click are performed. Can be asked. According to this invention,
The same operation as the touch panel can be performed on the display screen having no touch panel structure. In addition, as a sensor, a microphone that is already general-purpose, inexpensive, and easily available is used. As a result, the same effect as the above-described configuration of the invention using the camera unit as a sensor can be obtained. Further, for example, a configuration is known in which an operation is performed by a user's uttered voice, but in this case, for example, the user has an inconvenience that only the content related to the operation can be spoken during the operation. However, since the present invention uses GUI control based on a contact sound, the above-mentioned problem is solved, and the user can freely talk during the operation. In addition, the present invention that realizes a GUI operation based on this contact sound is used in combination with the above-described invention that realizes a GUI operation based on a point position obtained from a captured image, thereby providing a GUI operation with improved operability. You can also do it.

[Brief description of drawings]

FIG. 1 is a perspective view showing an outline of a pseudo touch panel operation system according to the present embodiment.

FIG. 2 is a block diagram showing an internal configuration example of a personal computer according to the present embodiment.

FIG. 3 is an explanatory diagram showing an operation procedure for setting frame information.

FIG. 4 is an explanatory diagram schematically showing a keystone correction process for frame information.

FIG. 5 is an explanatory diagram schematically showing a processing example for converting the in-image coordinates existing in the frame information into the actual coordinates on the display screen.

FIG. 6 is an explanatory diagram schematically showing a processing example for converting the in-image coordinates existing in the frame information into the actual coordinates on the display screen.

FIG. 7 is an explanatory diagram schematically showing a processing example for converting the in-image coordinates existing in the frame information into the actual coordinates on the display screen.

FIG. 8 is an explanatory diagram schematically showing an example of analysis processing for a contact sound.

FIG. 9 is an explanatory diagram showing a state transition example of a mouse left button operation according to an analysis result of a contact sound.

FIG. 10 is a block diagram showing a program structure example of an image / voice recognition driver.

FIG. 11 is a flowchart showing a processing operation according to an image / voice recognition driver corresponding to a pseudo touch panel operation.

FIG. 12 is a flowchart showing an example of pointer detection processing as processing according to the image recognition module.

FIG. 13 is an explanatory diagram showing a color area in a YUV chromaticity space.

FIG. 14 is a captured image captured by a camera unit,
It is explanatory drawing which shows the state in which the image part of the user's hand as a pointer has appeared.

FIG. 15 is an explanatory diagram showing another configuration example of the pseudo touch panel operation system of the present embodiment.

FIG. 16 is an explanatory diagram schematically showing a structural example of a touch panel.

[Explanation of symbols]

10 personal computer, 11 CPU, 17
Display monitor, 17a display screen, 18 HD
D, 18a image / speech recognition driver, 22 camera section, 23 (23A) microphone, 30 image / speech recognition module, 40 speech recognition module section, 41
Audio capture module, 42 audio analysis module, 50 image recognition module part, 51 video capture module, 52 coordinate acquisition module, 60 mouse interface application,
CR1 to CR4 Image corner designation cursor,

Continued front page    (72) Inventor Atsuhiko Imai             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation (72) Inventor Hirotomo Fukuda             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation F term (reference) 5B068 AA05 BB18 BC05 BD09 BD20                       BE08 CC06 CD05                 5B087 AA09 AB02 CC26 CC33 DD17                       DE07

Claims (15)

[Claims] 1. A display means for displaying and outputting an image on a display screen, an image capturing means for capturing a captured image captured by an image capturing apparatus, and the display in the captured image captured by the image capturing means. Image position recognition means for recognizing the image part area of the screen, and point position recognition for recognizing the position indicated by the pointer existing in the image part area recognized by the image part recognition means as the actual point position on the display screen. An information processing device comprising: a means; and an information processing means capable of executing required information processing according to the point position recognized by the point position recognition means. 2. A voice capturing means for capturing a voice signal collected by a microphone device, and a voice pointer captured by the voice capturing means, which is caused by a physical pointer coming into contact with the display screen. The extraction means for extracting the voice signal component of the contact sound and the analysis means for executing a required analysis process for the voice signal component of the contact sound extracted by the extraction means are provided, and the information processing means is the analysis means. The information processing apparatus according to claim 1, wherein the information processing apparatus is configured to be capable of executing required information processing based on the analysis result of. 3. At least an imaging device and an information processing device, wherein the imaging device is provided at a position where the entire display screen of the information processing device can be imaged, and the information processing device A display means for displaying and outputting an image, an image capturing means for capturing the captured image captured by the image capturing device, and an image partial area of the display screen in the captured image captured by the image capturing means. Image part recognition means, point position recognition means for recognizing a position pointed by a pointer existing in the image part area recognized by the image part recognition means as an actual point position on the display screen, and the point position recognition means And an information processing unit capable of executing required information processing according to the point position recognized by An information processing system to be. 4. The information processing system further includes a microphone device, and the information processing device includes a voice capturing unit that captures a voice signal collected by the microphone unit, and a voice capturing unit that captures the voice signal. Extraction means for extracting a voice signal component of a contact sound generated when a physical pointer comes into contact with the display screen from the voice signal, and a required audio signal component of the contact sound extracted by the extraction means. 4. The information processing apparatus according to claim 3, further comprising: an analysis unit that executes an analysis process, wherein the information processing unit is configured to be able to execute required information processing based on an analysis result of the analysis unit. Processing system. 5. An image capturing procedure for capturing a captured image captured by an image capturing apparatus arranged to capture a display screen of an information processing apparatus, and the display in the captured image captured by the image capturing procedure. Image part recognition procedure for recognizing the image part area of the screen, and point position recognition for recognizing the position pointed by the pointer existing in the image part area recognized by the image part recognition procedure as the actual point position on the display screen. A program for causing the information processing apparatus to execute a procedure, and an information processing procedure capable of executing required information processing according to the point position recognized by the point position recognition procedure. 6. A voice capturing procedure for capturing a voice signal picked up by a microphone device, and a physical pointer coming into contact with the display screen from the voice signal captured by the voice capturing procedure. The information processing device is caused to execute an extraction procedure for extracting a voice signal component of a contact sound, and an analysis procedure for performing a required analysis process for the voice signal component of the contact sound extracted by the extraction procedure, and the information described above. The program according to claim 5, wherein the processing procedure is configured to allow the information processing apparatus to execute required information processing based on an analysis result of the analysis procedure. 7. A display means for displaying and outputting an image on a display screen, a voice capturing means for capturing a voice signal picked up by a microphone device, and the voice signal captured by the voice capturing means, Extraction means for extracting an audio signal component of a contact sound generated when a physical pointer comes into contact with the display screen, and analysis for executing a required analysis process for the audio signal component of the contact sound extracted by the extraction means. An information processing apparatus comprising: means, and information processing means capable of executing required information processing based on an analysis result of the analysis means. 8. The information processing apparatus according to claim 7, wherein the analyzing unit is configured to analyze the strength level of the contact sound at a predetermined stage. 9. The information processing apparatus according to claim 7, wherein the analysis unit is configured to analyze temporal continuity of the contact sound. 10. At least a microphone device,
The information processing apparatus comprises a display means for displaying and outputting an image on a display screen, a voice capturing means for capturing a voice signal picked up by a microphone device, and the voice capturing means. Extraction means for extracting the sound signal component of the contact sound generated by the physical pointer coming into contact with the display screen from the captured sound signal, and the sound signal component of the contact sound extracted by the extraction means. An information processing system comprising: an analysis unit that executes required analysis processing; and an information processing unit that can execute required information processing based on an analysis result of the analysis unit. 11. The information processing system according to claim 10, wherein the analysis unit is configured to analyze the strength level of the contact sound in a predetermined stage. 12. The information processing system according to claim 10, wherein the analysis unit is configured to analyze temporal continuity of the contact sound. 13. A physical pointer comes into contact with a display screen of an information processing device from a voice capturing procedure for capturing a voice signal collected by a microphone device and the voice signal captured by the voice capturing procedure. Based on the analysis result of the above-mentioned analysis procedure and the extraction procedure for extracting the voice signal component of the contact sound caused by And an information processing procedure capable of executing required information processing by the information processing apparatus. 14. The program according to claim 13, wherein the analysis procedure is configured to analyze the strength level of the contact sound at a predetermined stage with respect to the contact sound. 15. The program according to claim 13, wherein the analysis procedure is configured to analyze temporal continuity of the contact sound.