JP2002318652A - Virtual input device and its program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an input interface suitable for a wearable computer to a user. SOLUTION: A virtual input device 1 is provided with a camera 3 for photographing a scenery in front of a user, and for acquiring the three- dimensional information of an object which is present in front of the user, a computer 5 for recognizing a face existing relatively near the user, and for superimposing the picture of a virtual input device corresponding to the recognized face on a picture outputted from the camera, and display devices 2a and 2b for displaying the picture on which the picture of the virtual input device is superimposed. The computer 5 detects the input operation of the user to the virtual input device displayed on the display devices 2a and 2b, and accepts the input from the user.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a virtual input device and a program capable of receiving an input from a user.

[0002]

2. Description of the Related Art With the miniaturization and high performance of computers, ultra-compact and lightweight computers called wearable computers have been researched and developed.

[0003] Higher precision and higher performance of a head mounted display (HMD) have been realized, and the possibility of practical use and spread of wearable computers has been increasing.

[0004]

The above wearable computer needs an input interface for inputting contents such as characters and pictures. As this input interface, addition of special input hardware, assistance of a conventional GUI input interface, or voice input can be considered.

[0005] However, the content input interface available on the wearable computer is still poor.

[0006] The present invention has been made in view of the above-described problems, and has as its object to provide a virtual input device capable of providing a user with an input interface suitable for a wearable computer.

[0007]

According to the present invention, there is provided a virtual input device for recognizing a surface relatively close to a user, and for providing the user with a virtual input interface corresponding to the recognized surface. And thereby the above object is achieved.

According to another aspect of the present invention, there is provided a virtual input device for photographing a scene in front of a user and acquiring three-dimensional information of an object existing in front of the user. A computer that recognizes a surface that is relatively close to the user based on the image, and superimposes an image of a virtual input device corresponding to the recognized surface on an image output from the camera; and an image of the virtual input device. And a display device that displays an image on which the image is superimposed. The computer receives an input from the user by detecting an input operation of the user on the virtual input device displayed on the display device. Thereby, the above object is achieved.

A virtual input device according to the present invention includes a camera for photographing a scene in front of a user, a sensor for detecting the intensity of reflected light from an object existing in front of the user, and the detected reflected light. A computer that recognizes a surface relatively close to the user based on the strength of the user, and superimposes an image of a virtual input device corresponding to the recognized surface on an image output from the camera; and A display device that displays an image on which an image of the input device is superimposed, wherein the computer detects an input operation of the user on the virtual input device displayed on the display device, thereby inputting from the user. Accept. Thereby, the above object is achieved.

[0010] The virtual input device may be one selected from a plurality of input devices including a keyboard and a handwriting input pad.

[0011] The display device may be a head mounted display (HMD) that can be mounted on the head of the user.

A program according to the present invention is a program for causing a computer to execute a virtual input process, wherein the virtual input process includes a step of recognizing a surface relatively close to a user; Providing the user with a virtual input interface according to the above, whereby the above object is achieved.

Another program according to the present invention is connected to a camera for photographing a scene in front of the user, acquiring three-dimensional information of an object existing in front of the user, and a display device for displaying an image. A program for causing a computer to execute a virtual input process, wherein the virtual input process recognizes a surface relatively close to the user based on the acquired three-dimensional information; Superimposing an image of the virtual input device according to the recognized surface on an image output from the camera; displaying an image on which the image of the virtual input device is superimposed on the display device; and Receiving input from the user by detecting an input operation of the user with respect to the virtual input device displayed in; And encompasses, thereby the objective described above being achieved.

According to another program of the present invention, there is provided a camera for photographing a scene in front of a user, a sensor for detecting the intensity of light reflected from an object existing in front of the user, and a display device for displaying an image. And a computer connected to the virtual input processing, the virtual input processing, based on the intensity of the detected reflected light, a surface that is relatively close to the user, Recognizing, superimposing an image of the virtual input device corresponding to the recognized surface on an image output from the camera, and displaying the image on which the image of the virtual input device is superimposed on the display device And accepting an input from the user by detecting an input operation of the user on the virtual input device displayed on the display device. Tsu and includes a flop, thereby the objective described above being achieved.

[0015]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1A shows a configuration of a virtual input device 1 according to an embodiment of the present invention.

The virtual input device 1 includes a head-mounted display (HMD) 2 that can be mounted on the user's head, a camera 3 that captures a scene in front of the user, and reflected light from an object existing in front of the user. Sensor 4 for detecting the strength of
And a computer 5 connected to the HMD 2, the camera 3, and the sensor 4.

The computer 5 is attached to the user's waist using a belt, for example. The computer 5 is preferably a small and lightweight computer so that it can be easily carried. In the example shown in FIG.
2. The camera 3, the sensor 4, and the computer 5 are connected by wire. However, it goes without saying that the HMD 2, the camera 3, the sensor 4, and the computer 5 may be wirelessly connected. The computer 5 is H
It may be attached to the MD2 or may be built in the HMD2.

In the example shown in FIG. 1A, the HMD 2 is a glasses-type HMD. The HMD 2 includes a display unit 2a for the right eye and a display unit 2b for the left eye, instead of a lens of ordinary glasses. An image is displayed on the inner surface (the surface closer to the user) of the display units 2a and 2b based on the signal output from the computer 5. This allows the user to
Instead of a real scene, a virtual scene generated by the computer 5 can be viewed. Note that HMD2
As any type of HMD besides the glasses type HMD
MD can be used.

The camera 3 and the sensor 4 are connected to the HMD so that the optical axis 3a of the camera 3 and the optical axis 4a of the sensor 4 are parallel.
2 (see FIG. 1B). This is camera 3
This is because the sensor 4 and the sensor 4 need to face the same direction with respect to the user.

The camera 3 and the sensor 4 may be formed integrally. When the camera 3 and the sensor 4 are integrally formed, it is preferable to design the camera 3 so that the optical axis 3a of the camera 3 and the optical axis 4a of the sensor 4 coincide.

The camera 3 captures a scene in front of the user. Here, in this specification, “in front of the user” means that the front direction of the user is the reference direction 6 and is within a range of a predetermined angle from the reference direction 6 (for example, when the reference direction 6 is 0 degree,
(In the range of −α degrees to + α degrees) (see FIG. 1C). The optical axis 3a of the camera 3 (FIG. 1B) and the optical axis 4a of the sensor 4 (FIG. 1B) preferably coincide with the reference direction 6.

[0023] The camera 3 is preferably a digital video camera. Alternatively, an analog video camera may be used as the camera 3. In this case, the computer 5 may convert an analog signal output from the camera 3 into a digital signal.

The sensor 4 is, for example, an image sensor that emits near-infrared light from an LED and captures the reflected light via a lens. As such an image sensor, for example, a URL “http: //mptech.i” is used.
nfortech. co. jp "website (trade name: PC motion capture device" Motion Processor ").

FIG. 2 shows an example in which reflected light reflected by an object (hand) in front of the image sensor is imaged through a lens. The image sensor detects the intensity of light reflected from the object. By detecting the intensity of light reflected from the object, the distance between the image sensor and the object can be detected. This is because the relationship between the intensity I of the reflected light and the distance D between the image sensor and the object is represented by (Equation 1).

[0026]

I = k / D ² where k is a constant.

(Equation 1) indicates that the distance D is small (ie,
When the object is near, the intensity of the reflected light is large, and the intensity of the reflected light sharply decreases as the distance D increases (that is, as the object increases). By utilizing such a relationship between the distance D and the intensity I of the reflected light, the background can be easily removed from the image captured by the image sensor. As a result, it is possible to recognize a surface relatively close to the user.

Here, in this specification, "relatively close to the user" refers to a range where a part of the user's body (typically, the user's hand) can reach. As described below,
This is because the present invention intends that the user performs an input operation on the virtual input device displayed in the virtual environment. In this specification, “surface recognition” is used.
“Detecting” means detecting at least one of the inclination of the surface, the size of the surface, and the distance between the surface and the user. In the embodiment, an example in which the plane is a plane will be described, but the plane is not limited to the plane. The surface may be any surface as long as it has a shape that allows the virtual input device to be superimposed and displayed without any inconvenience.

Next, with reference to FIGS. 3 and 4, the principle of detecting the size of a plane relatively close to the user and substantially parallel to the user using the above-described image sensor will be described. explain.

FIG. 3 shows a case where the distance from the image sensor is 50 cm.
Shows an example of an image obtained as a result of imaging reflected light from a small piece placed at the position of FIG. As shown in FIG. 3, a portion 30 (shown by hatching) corresponding to a small piece is detected as a black image (that is, an image with high luminance). As described above, a plane existing relatively close to the image sensor is detected as an image with high luminance.

FIG. 4 shows a luminance distribution along the line XX 'shown in FIG. From FIG. 4, the portion 4 corresponding to the small piece
0 indicates that the luminance difference is about 80 or more compared to other parts.

The XX 'line shown in FIG. 3 is scanned in a direction perpendicular to the XX' line, and a portion where the luminance difference is equal to or more than a predetermined value (for example, 80) as compared with other portions for each scan. By obtaining the length of the image sensor and integrating the lengths, the size of a plane relatively close to the image sensor (that is, a plane relatively close to the user) can be detected. Such detection is performed by the computer 5 based on the output from the sensor 4.

For example, when there is a user's hand (a hand with all fingers open) immediately before the sensor 4, the computer 5 determines the size of the user's hand as “the user is relatively close to the user. As the size of the plane to perform. Further, for example, when there is a wall surface immediately before the sensor 4, the computer 5 detects the size of the wall surface as "the size of a plane relatively close to the user".

It should be noted that the image output from the above-described image sensor has depth information in a pseudo manner (that is, the above-described image sensor acquires three-dimensional information of a target object from the viewpoint of the user). Is possible). Therefore, by performing relatively simple processing without performing complicated image processing, it becomes possible to recognize a plane such as a desk surface that is not always parallel to the user.

For example, assume that there is a desk near the user, and books are stacked on the desk. In this case, the computer 5 recognizes a vacant (unbooked) plane that can be displayed by the virtual input device as a “plane that is relatively close to the user”.

As a method for acquiring three-dimensional information of an object, an optical radar method, an optical projection method, a binocular stereo method, and the like are known. However, each of these methods has a limited environment, requires expensive equipment, and requires a long calculation time. By using the image sensor as described above, it does not require expensive equipment, does not depend on the environment,
Image processing can be performed at a relatively high speed.

FIG. 5 shows the internal configuration of the computer 5. The computer 5 includes a CPU 51, a memory 52,
Frame buffer 53 and interface 5 for camera 3
4, an interface 55 for the sensor 4, and an interface 56 for the HMD 2. The components included in the computer 5 are connected to each other via a bus 57.

The memory 52 stores a program for causing the computer 5 to execute virtual input processing. This program may be stored in the memory 52 in advance when the computer 5 is shipped. Alternatively, this program may be stored in the memory 52 after the computer 5 is shipped. For example, the user may download this program from a specific website on the Internet for a fee or free of charge, and install the downloaded program on the computer 5.

FIG. 6 shows the procedure of the virtual input process. The execution of this virtual input process is controlled by the CPU 51 (FIG. 5). Hereinafter, each step of the virtual input process will be described in detail.

Step S1: The image output from the camera 3 is input to the computer 5. The image input to the computer 5 is stored in the frame buffer 53 via the camera 3 interface 54.

Step S2: The signal output from the sensor 4 is input to the computer 5. The signal input to the computer 5 is stored in the memory 52 via the sensor 4 interface 55. The signal output from the sensor 4 indicates the intensity of reflected light from an object existing in front of the user.

Step S3: The CPU 51 detects the size of a plane relatively close to the user based on the signal output from the sensor 4. For such detection, for example, the XX ′ line shown in FIG. 3 is scanned in a direction perpendicular to the XX ′ line, and the brightness difference is a predetermined value (for example, 80) This is done by finding the length of a certain portion and integrating the length.

Step S4: The CPU 51 proceeds to step S4.
According to the size of the plane detected in step 3, one of a plurality of virtual input devices prepared in advance is selected. Multiple virtual input devices are "handwriting input pads"
And "keyboard". Here, the “handwriting input pad” refers to an input device that provides a user with an input interface for inputting characters (including numbers and symbols) by handwriting, and the “keyboard” refers to pressing a typeface. An input device for inputting characters (including numbers and symbols) by performing an input operation. Information defining the virtual input device (for example, information indicating functions and operation methods of the plurality of virtual input devices) and a selection rule used to select one from the plurality of virtual input devices are stored in the memory 52 in advance. Is stored.

For example, if the size of the plane detected in step S3 is equal to or smaller than a predetermined threshold, the CPU
51 selects a “handwriting input pad” from a plurality of virtual input devices. This is because the "handwriting input pad" can be used even on a narrow flat surface. If the size of the plane detected in step S3 is larger than the predetermined threshold, the CPU 51 selects a “keyboard” from the plurality of virtual input devices. The "keyboard" is not suitable for use on a narrow flat surface.

It goes without saying that the selection of the input device is not limited to the alternative. The number of input devices may be three or more.

Step S5: The CPU 51 proceeds to step S5.
4 to the image output from the camera 3 (ie, the frame buffer 5).
3 (image stored in 3). Such superimposition processing is achieved, for example, by changing the luminance value of a portion of the image stored in the frame buffer 53 that overlaps the image of the virtual input device to the luminance value of the image of the virtual input device.

Step S6: The image stored in the frame buffer 53 is output from the computer 5 via the HMD2 interface 56. Thereby, the image on which the image of the virtual input device is superimposed is displayed on the HMD 2.

Step S7: The CPU 51 detects a user's input operation on the virtual input device displayed on the HMD 2. Such detection can be performed, for example, by using a technique that tracks the movement of the user's hand or finger captured by the camera 3. As a technique for tracking the movement of the user's hand or finger, various techniques using image processing can be applied. For example, the technology described in the following document is an example of a technology that can be suitably applied as a technology for tracking the movement of a user's hand or finger.

Reference: Sasaki et al., "Tenohiradaira:
"Input Interface for Wearable Computer", IEICE Technical Report (TECNICAL REPORT
OFIICE) PRMU 2000-157 (20
01-1) Alternatively, a contact sensor may be attached to the user's fingertip, and the movement of the user's finger may be tracked based on an output signal from the contact sensor.

As described above, the virtual input device 1 is connected to the HMD 2
The input from the user is received by detecting the user's input operation on the virtual input device displayed in (1).

FIG. 7 shows a state where the user is performing an input operation on the “handwriting input pad” in the virtual environment. The example shown in FIG. 7 is an example in which it is detected that the user's hand is located immediately before the user. In this case, a “handwriting input pad” suitable for the size of the user's hand is displayed in the virtual environment. The virtual input device 1 allows the user to perform an input operation on the “handwriting input pad”.

FIG. 8 shows a state where the user is performing an input operation on the “keyboard” in the virtual environment. The example illustrated in FIG. 8 is an example in which it is detected that a wall surface exists immediately before the user. In this case, a “keyboard” suitable for the size of the wall is displayed in the virtual environment. The virtual input device 1 allows a user to perform an input operation on a “keyboard”.

If it is detected that there is a desk immediately before the user, an input interface (for example, a "handwriting input pad" or a "keyboard") suitable for the size of an empty area of the desk is used. May be provided to the user.

According to the virtual input device 1, an input interface usable in the virtual environment is switched according to the size of the surface relatively close to the user. This makes it possible to provide the user with an input interface suitable for a wearable computer.

As described above, the sensor 4 and the computer 5 function as “means for recognizing a surface relatively close to the user”. Further, the HMD 2, the camera 3, and the computer 5 function as “means for providing a user with a virtual input interface corresponding to the recognized surface”.

Note that the camera 3 captures an image of the scene in front of the user, and the object 3 existing in front of the user.
If you have the function of acquiring dimensional information,
The sensor 4 can be omitted from the configuration of the virtual input device 1. In this case, the computer 5 may recognize a surface relatively close to the user based on the three-dimensional information acquired by the camera 3. Therefore,
In this case, the camera 3 and the computer 5 function as “means for recognizing a surface that is relatively close to the user”. For example, a stereo camera has a function of photographing a scene in front of a user and acquiring three-dimensional information of an object existing in front of the user from parallax of an edge portion at an end of a region. Therefore, camera 3
By using a stereo camera as the sensor, the sensor 4 becomes unnecessary.

Next, consider the case where the plane on which the virtual input device is to be superimposed is inclined with respect to the optical axis 4a of the sensor 4.

When an object is irradiated with infrared rays, the reflected light from the object is composed of a diffuse reflection component and a specular reflection component.

The diffuse reflection component is light reflected at the same intensity in any direction irrespective of the incident angle. When a point light source is used, it follows the inverse square law of distance and Lambert's cosine law. If the light amount of the light source is L, the reflected light amount is I _d , the distance to the object is r, and the incident angle is α, (FIG. 9) and (Equation 2) hold.

[0060]

I _d = R _d (L / r ² ) cos (α) where R _d is the diffuse reflectance.

The specular reflection component is light reflected in the direction of the reflection angle equal to the incident angle. However, when the light is not perfectly specularly reflected, light is also reflected in the vicinity of the regular reflection direction. Here, using the model of Phong, the rate of decrease of the specular reflection component due to the deviation angle γ between a sight line and a specular reflection direction cos ⁿ
Approximate by γ. In this case, the amount of reflected light and I _s, (10), equation (3) holds.

[0062]

I _s = R _s (L / r ² ) cos ⁿ γ where R _s is the specular reflectance, γ is the angle with the line of sight in the regular reflection direction, and n is the highlight characteristic.

Accordingly, the intensity I of the reflected light when the two components of the diffuse reflection component and the specular reflection component are taken into account is represented by (Equation 4).

[0064]

## EQU4 ## I = R _d (L / r ² ) cos (α) + R _s (L /
From r ² ) cos ⁿ γ (Equation 4), it can be seen that the intensity of the reflected light is not simply inversely proportional to the square of the distance from the light source. Sensor 4
Considering a plane perpendicular to the optical axis 4a, the value of α is small near the center of the plane, so that not only the diffuse reflection component but also the specular reflection component becomes large, and the image becomes extremely bright. Conversely, since the value of the specular reflection component hardly appears in the peripheral portion of the plane, it becomes extremely dark.

As described above, since the image output from the sensor 4 has depth information, three-dimensional information of the object can be obtained. The computer 5
Recognize a plane relatively close to the user based on the dimensional information. However, the pixel value of the three-dimensional information gradually becomes darker from the center to the periphery of the plane due to the reflection characteristics taking into account the diffuse reflection component and the specular reflection component. Therefore, the computer 5 converts the pixel value of the three-dimensional information so that the pixel value changes along the inclination of the plane.
By performing such a conversion, it is possible to determine the equation of the plane from the inclination of the plane and three appropriate points on the plane after the conversion.

Here, consider the case where the plane is inclined with respect to the optical axis 4a of the sensor 4 (for example, the case where the plane is inclined by β with respect to the light source as shown in FIG. 11).

In FIG. 11, d indicates the distance from the sensor 4 to the plane, and point p ₁ indicates the center of the image. The distance from the light source Q to the point p ₂ can be expressed as d / cos (α + β). Therefore, d / cos (α +
β) may be substituted. Since the sensor 4 is located at the same position as the light source, γ can be expressed as 2α. Therefore,
The reflected light when the plane is inclined by β with respect to the light source is
It is represented by (Equation 5).

[0068]

[Equation 5] I (α) = (L / r ² ) ｛R _d cos (α) + R
By multiplying the input image by (Equation 6) according to _s cos ⁿ (2α)｝ (Equation 5), the image output from the sensor 4 is converted. Thereby, the equation of the plane can be determined.

[0069]

F (α) = {βα + I (0)} / I (α) In the above-described embodiment, an example has been described in which an image is displayed on a head mounted display (HMD) of a stereotype display. However, the present invention is not limited to this. Instead of a stereotype head mounted display (HMD), any type of head mounted display (HMD) can be used. For example, a video see-through type HMD (for example, EyeTrek manufactured by Olympus Corp.) which does not allow the foreground to pass through like a pair of glasses and can only see the screen display when worn may be used, or an optical system which superimposes the screen display on the transmitted outside scenery A see-through type HMD (for example, a product manufactured by Seibner, a product manufactured by Micro Optical, or a glasstron manufactured by SONY) may be used.

[0070]

According to the present invention, it is possible to provide a virtual input device capable of providing a user with an input interface suitable for a wearable computer.

[Brief description of the drawings]

FIG. 1A is a diagram showing a configuration of a virtual input device 1 according to an embodiment of the present invention.

FIG. 1B is a diagram showing a side surface of the virtual input device 1 according to the embodiment of the present invention;

FIG. 1C is a diagram illustrating an upper surface of the virtual input device 1 according to the embodiment of the present invention;

FIG. 2 is a diagram illustrating an example in which reflected light reflected by an object (hand) in front of an image sensor is imaged through a lens.

FIG. 3 is a diagram showing an example of an image obtained as a result of imaging reflected light from a small piece placed at a distance of 50 cm from an image sensor.

FIG. 4 is a diagram showing a luminance distribution along the line XX ′ shown in FIG. 3;

FIG. 5 is a diagram showing an internal configuration of a computer 5;

FIG. 6 is a diagram showing a procedure of a virtual input process.

FIG. 7 is a diagram illustrating a state in which a user performs an input operation on a “handwriting input pad” in a virtual environment;

FIG. 8 is a diagram illustrating a state in which a user performs an input operation on a “keyboard” in a virtual environment.

FIG. 9 is a diagram for explaining a reflection diffusion model;

FIG. 10 is a diagram for explaining a specular reflection model;

FIG. 11 is a diagram illustrating a case where a plane is inclined by β with respect to a light source;

[Explanation of symbols]

 Reference Signs List 1 virtual input device 2 head mounted display (HMD) 3 camera 4 sensor 5 computer

Continuation of the front page (72) Inventor Masaki Chikuma 8916-5 Takayamacho, Ikoma City, Nara Prefecture Student Residence Hall 1-207 (72) Inventor Tatsuyuki Kawamura 728-501 Iwagamicho, Nakagyo-ku, Kyoto, Kyoto (72) Inventor Taniguchi Yuichiro 8916-5 Takayamacho, Ikoma City, Nara Prefecture Student Residence 4-106 F-term (reference) 5B020 CC02 CC06 CC12 DD30 FF51 GG13 5B087 AA09 AE00 BC04 BC12 BC13 BC26 BC32 DE02

Claims (8)

[Claims] 1. A virtual input device comprising: means for recognizing a surface relatively close to a user; and means for providing the user with a virtual input interface corresponding to the recognized surface. 2. A camera for photographing a scene in front of the user and acquiring three-dimensional information of an object existing in front of the user; and a camera of the user based on the acquired three-dimensional information. A computer that recognizes a nearby surface and superimposes an image of a virtual input device corresponding to the recognized surface on an image output from the camera; and displays an image on which the image of the virtual input device is superimposed. A display device, wherein the computer receives an input from the user by detecting an input operation of the user on the virtual input device displayed on the display device,
Virtual input device. A camera for photographing a scene in front of the user; a sensor for detecting the intensity of reflected light from an object existing in front of the user; and a sensor for detecting the intensity of the reflected light. A computer for recognizing a surface relatively close to the user and superimposing an image of a virtual input device corresponding to the recognized surface on an image output from the camera; and A display device that displays the displayed image, wherein the computer receives an input from the user by detecting an input operation of the user on the virtual input device displayed on the display device,
Virtual input device. 4. The virtual input device according to claim 2, wherein the virtual input device is one selected from a plurality of input devices including a keyboard and a handwriting input pad. 5. The virtual input device according to claim 2, wherein the display device is a head-mounted display (HMD) that can be mounted on a head of the user. 6. A program for causing a computer to execute a virtual input process, the virtual input process comprising: a step of recognizing a surface existing relatively close to a user; and a virtual process corresponding to the recognized surface. Providing an input interface to the user. 7. A virtual input process to a computer connected to a camera that captures a scene in front of a user and acquires three-dimensional information of an object existing in front of the user, and a display device that displays an image. The virtual input processing includes: a step of recognizing a surface relatively close to the user based on the obtained three-dimensional information; Superimposing an image of the virtual input device on the image output from the camera; displaying an image on which the image of the virtual input device is superimposed on the display device; and displaying the virtual image displayed on the display device. Receiving an input from the user by detecting an input operation of the user on the input device.
program. 8. A computer connected to a camera for photographing a scene in front of the user, a sensor for detecting the intensity of reflected light from an object present in front of the user, and a display device for displaying an image. A program for causing the virtual input process to execute a virtual input process, wherein the virtual input process recognizes a surface relatively close to the user based on the detected intensity of the reflected light; Superimposing an image of the virtual input device according to the recognized surface on an image output from the camera; displaying an image on which the image of the virtual input device is superimposed on the display device; and the display device. Receiving an input from the user by detecting an input operation of the user on the virtual input device displayed on the virtual input device. Lamb.