WO2006004155A1

WO2006004155A1 - 3d coordinate input device using terminal device

Info

Publication number: WO2006004155A1
Application number: PCT/JP2005/012488
Authority: WO
Inventors: Akihiko Iketani; Noboru Nakajima
Original assignee: Nec Corporation
Priority date: 2004-07-07
Filing date: 2005-07-06
Publication date: 2006-01-12
Also published as: JP2007310424A

Abstract

A 3D coordinate input device is mounted on a terminal device and includes a moving picture imaging unit, a 3D operation calculation unit, an operation information generation unit, and a display control unit. The 3D operation calculation unit calculates movement of the terminal device in a 3D space as a change of 3D coordinate according to the moving picture imaged by the moving picture imaging unit. According to the calculated movement of the terminal device, the operation information generation unit generates operation information for modifying the image displayed on display means for displaying an image. According to the operation information generated, the display control unit controls the display unit according to predetermined software for modifying the image displayed on the display unit.

Description

Specification

Three-dimensional coordinate input device using terminal device

Technical field

The present invention relates to a three-dimensional coordinate input device, and more particularly to a three-dimensional coordinate input device using a camera capable of shooting a moving image.

Background art

[0002] A three-dimensional coordinate input device for performing three-dimensional coordinate input by mounting a mechanism for inputting a z-coordinate in a mouse for inputting a normal xy coordinate is disclosed in Japanese Patent Laid-Open No. 2001-360252. JP-A-11-134111 and JP-A-8-179883 have proposed.

[0003] A trackball is mounted on a normal mouse, the inclination of the plane in the three-dimensional space is specified by the trackball, and the two-dimensional coordinates on the same plane are specified by the mouse. A three-dimensional coordinate input device in which coordinate input is performed is also proposed by Japanese Patent Laid-Open Nos. 9-26851 and 1-100621.

[0004] On the other hand, an inertial sensor corresponding to each dimension of xyz is arranged on a terminal, and the movement of the terminal is read from the detection result of each inertial sensor so that three-dimensional coordinate input is performed. — Proposed by Japanese Patent No. 98630 and Japanese Patent Laid-Open No. 1-96720.

[0005] However, the above-described conventional technique has the following problems.

[0006] First, in a device in which a z-coordinate input mechanism is mounted on a mouse, the device itself becomes a large force.

[0007] Secondly, it is difficult to input coordinates intuitively in a device in which a z-coordinate input mechanism is mounted on a mouse. The reason is that the user needs to operate both the z-coordinate input mechanism and the mouse at the same time.

[0008] Thirdly, in an apparatus in which the three-dimensional coordinates of the terminal are acquired by the sensing result of the inertial sensor

This means that the terminal is expensive and expensive.

[0009] Fourthly, in a device in which the three-dimensional coordinates of the terminal are obtained from the sensing result of the inertial sensor. Even if the user is stationary, if the user is on a moving object such as a car or train, the operation is input to the software against the user's intention. The reason is that the inertial sensor detects the absolute movement of the terminal rather than the relative movement between the user and the terminal.

Disclosure of the invention

Problems to be solved by the invention

[0010] An object of the present invention is to solve the above-described problems and without newly adding a special sensor.

An object of the present invention is to provide a three-dimensional input device capable of realizing intuitive three-dimensional coordinate input. Means for solving the problem

[0011] According to the present invention, a three-dimensional coordinate input device mounted on a terminal device is based on a moving image capturing unit that captures a moving image and a moving image captured by the moving image capturing unit. The three-dimensional motion calculation means for calculating the movement of the terminal device in the three-dimensional space as a change in three-dimensional coordinates, and the display means for displaying the image based on the calculated movement of the terminal device. An operation information generating means for generating operation information for changing the displayed image, and an image displayed on the display means is changed according to the generated operation information, and the display means is controlled based on predetermined software A three-dimensional coordinate input device having display control means is provided.

[0012] According to the present invention, since the three-dimensional coordinate input is performed based on the captured image of the moving image capturing means installed in the terminal device, the low-cost and intuitive tertiary can be achieved without adding a special sensor. An original coordinate device can be realized.

In addition, according to the present invention, when inputting three-dimensional coordinates, it is only necessary to move the terminal device in a three-dimensional space where it is not necessary to operate a plurality of mechanisms simultaneously. Can be done.

[0014] Further, according to the present invention, it is possible to detect relative movement between the user who operates the terminal device or the moving body on which the user is riding and the terminal device, and the user gets on the moving body. Even in such a case, it is possible to prevent an operation instruction based on software from being input against the user's intention.

[0015] The present invention provides viewers such as images and documents on a mobile terminal, Web pages, and browser operations. It can be applied to applications such as user interface for work.

[0016] The present invention can also be applied to uses such as a user interface for operating characters around the virtual world in a game.

Brief Description of Drawings

FIG. 1 is a block diagram of a three-dimensional coordinate input device according to a first embodiment of the present invention.

FIG. 2 is an external view of a terminal device equipped with the three-dimensional coordinate input device according to the first embodiment.

FIG. 3 is a flowchart showing the operation of the three-dimensional coordinate input apparatus according to the first embodiment.

FIG. 4 is an external view of a terminal device equipped with a three-dimensional coordinate input device according to a second embodiment of the present invention.

FIG. 5 is an external view of a terminal device equipped with a three-dimensional coordinate input device according to a third embodiment of the present invention.

FIG. 6 is a block diagram of a three-dimensional coordinate input device according to a fourth embodiment of the present invention.

FIG. 7 is a diagram showing a three-dimensional coordinate system.

FIG. 8 is a diagram showing how three-dimensional motion is calculated.

[Fig. 9] Fig. 9 is a diagram showing how three-dimensional motion is calculated.

FIG. 10 is a diagram showing how the display image is changed based on the operation information.

FIG. 11 is a diagram showing how the display image is changed based on the operation information.

BEST MODE FOR CARRYING OUT THE INVENTION

[0018] First Embodiment

Referring to FIG. 1, the three-dimensional coordinate input device according to the first embodiment of the present invention includes a camera 11, a three-dimensional motion calculation unit 12, an operation information generation unit 13, a display control unit 14, an operation information enable switch 15, Have

As shown in FIG. 2, the camera 11 is installed on the opposite side of the casing of the terminal device 20 from the side on which the display unit 21 is installed! RU 3D motion calculator 12 and operation information generator 13 The control unit 14 is accommodated in the casing of the terminal device 20. As the display unit 21, a display device for displaying various information such as an LCD (Liquid Crystal Display) is used.

[0020] The camera 11 captures the subject 30, and for example, a digital camera having a function of capturing a moving image is used. Note that the camera 11 may include a light source such as a flash device or a light for capturing the subject 30 clearly. Here, the subject 30 may be a dedicated subject used when the terminal device 30 is used, such as a landscape around the terminal device 20 or an object on which a texture such as a grid or a character is printed.

[0021] The three-dimensional motion calculation unit 12 changes the three-dimensional coordinates of the three-dimensional motion of the terminal device 20 generated by the user 40 moving the terminal device 20 from the moving image obtained by photographing the subject 30 with the camera 11. Calculate as

The operation information generation unit 13 converts the three-dimensional motion calculated by the three-dimensional motion calculation unit 12 into operation information for changing the image displayed on the display unit 21.

[0023] The display control unit 14 controls the display unit 21 based on predetermined software (for example, a Web browser), and changes the image displayed on the display unit 21 in accordance with the generated operation information.

The operation information enabling switch 15 outputs a control signal for enabling or disabling the operation information generated by the operation information generating unit 13 to the operation information generating unit 13 according to the operation of the user 40.

Next, the operation of the three-dimensional coordinate input apparatus of the present embodiment will be described with reference to the flowchart of FIG.

[0026] First, in step 101, the three-dimensional motion calculation unit 12 performs a three-dimensional movement of the terminal device 30 caused by the user 40 moving the terminal device 20 from a moving image obtained by photographing the subject 30 with the camera 11. Calculate

[0027] Next, in step 102, the operation information generation unit 13 converts the three-dimensional motion calculated by the three-dimensional motion calculation unit 12 into operation information for changing the image displayed on the display unit 21. Replace.

Next, in step 103, the operation information generating unit 13 refers to the operation information enabling switch 15, and when the switch 15 is on, the operation information is displayed in step 104 in the display control unit 14. input. When switch 15 is off, no operation information is entered.

[0029] Finally, in step 105, the display control unit 14 changes the image displayed on the display unit 21 in accordance with the generated operation information.

[0030] In the present embodiment, since the three-dimensional coordinates are calculated from the moving image captured by the camera 11 mounted on the terminal device 20, a compact and inexpensive device that does not require a new camera to be added to the terminal device 20 is required. A three-dimensional coordinate input device can be realized.

[0031] Further, in this embodiment, since a user who does not need to operate a plurality of mechanisms at the same time moves the terminal device 20 in the space as a three-dimensional coordinate input, it is intuitive. Coordinate input is possible.

[0032] Further, according to the present embodiment, since the relative motion between the user himself or a moving body (automobile or the like) on which the user is riding and the terminal device 20 can be detected, the user can move the moving body. Even if you are on a vehicle, you can prevent unexpected operation information from being generated due to the movement of the moving object, and prevent unintended operation information from being input to the software. It is out.

[0033] When the display unit 21 has a backlight, it is possible to stably illuminate the face of the user 40 as a subject, and the three-dimensional motion can be calculated stably even in a dark place. .

[0034] Second Embodiment

The configuration itself of the three-dimensional coordinate input device according to the present embodiment is the same as that of the above-described three-dimensional coordinate input device according to the first embodiment, but in this embodiment, as shown in FIG. In order to use the face as a subject, the camera 11 is installed on the same side as the display side 21 of the terminal device 20 is installed.

[0035] Also in the present embodiment, the camera 11 may include a light source such as a flash device or a light for clearly photographing the user 40 as a subject. Alternatively, the backlight of the display unit 21 attached to the user 40 may be used as the light source.

In the present embodiment, since the camera 11 is installed so as to photograph the face of the user 40, the subject 30 can always be captured within a certain distance from the camera 11, and is suitable around the user 40. Even when a sharp subject does not exist, 3D motion can be calculated stably.

[0037] Third Embodiment

As shown in FIG. 5, the three-dimensional coordinate input device according to the present embodiment includes the first and second cameras in that the terminal device 20 has cameras 11 that capture the face of the subject 30 and the user 40, respectively. This is different from the three-dimensional coordinate input device according to the embodiment.

Therefore, according to the present embodiment, even in a situation where only one of the user 40 and the subject 30 can be photographed, the three-dimensional motion can be calculated stably. In addition, when both the user 40 and the subject 30 can be photographed, the accuracy of the three-dimensional motion is improved.

[0039] Fourth Embodiment

In this embodiment, a three-dimensional coordinate input program, which is a processing card executed in each part of the three-dimensional motion calculation unit 12, the operation information generation unit 13, and the display control unit 14 in FIG. The three-dimensional coordinate input program is executed by a computer mounted on the terminal device 20 in advance.

[0040] Fifth Embodiment

Referring to FIG. 6, in the three-dimensional coordinate input apparatus according to the present embodiment, the processing of the three-dimensional motion calculation unit 12 and the operation information generation unit 13 is executed by the computer 16, and the display control unit 14 and the display unit 21 are processed by the computer 17. Executed by. It is assumed that the computer 16 and the computer 17 are connected to each other through a serial or parallel interface such as USB (Universal Serial Bus). Also, the computer 16 and the computer 17 may be connected via a communication network such as force Ethernet (registered trademark).

[0041] The operation of the present embodiment is the same as that of the first embodiment. Note that the attachment position of the camera 11 in this embodiment is the same as any of the attachment positions in the first to third embodiments described above.

This embodiment also has the same advantages as the first to third embodiments described above.

[0043] The terminal device 20 includes a PDA (Personal Digital Assistant) having a camera function, a mobile phone terminal, a personal computer, a portable game machine, and the like. Further, the terminal device 20 may be a device that can be connected to a communication network or a device that cannot be connected to a communication network. Next, a method for calculating the three-dimensional motion in the above three-dimensional coordinate input apparatus will be described in detail.

The example described below corresponds to the second embodiment described above. Here, a case will be described in which the above-described three-dimensional coordinate input device is used as a three-dimensional coordinate input device that operates an image viewer operating on a digital camera.

FIG. 7 is an explanatory diagram showing a three-dimensional coordinate system. As shown in Fig. 7, in this example, the origin 200 is the intersection of the optical axis of the digital camera and the image plane, and the X-axis 201 and y-axis 202 are made to coincide with the X-axis and y-axis of the digital camera image plane, respectively. . The z-axis 203 is an axis that passes through the origin 200 and is orthogonal to the x-axis 201 and the y-axis 202, and the direction in which the subject exists is positive.

[0047] The three-dimensional motion calculation unit 12 calculates the three-dimensional motion of the digital camera based on each frame of the moving image photographed by the digital camera. Here, the three-dimensional motion consists of the amount of parallel movement of the digital camera in the x, y and z axis directions and the amount of rotation around each axis. In this example, literature: Ikeya, Nakajima, Sato, Ikeda, Kambara, Yokoya, Yamada, “Video mosaicing and super-resolution by camera path estimation for paper”, IEICE Technical Report TL2003—37 PRMU2003—233 , February 19, 2004, pp. 49-54 shall be used to calculate 3D motion.

[0048] In the three-dimensional motion calculation method described in Reference 1, first, a point where edges in a plurality of directions called feature points intersect is obtained on the image of the previous frame. Next, for each feature point, the corresponding coordinates in the current frame are found by tracking processing using template matching. Then, the three-dimensional coordinates of each feature point obtained in the previous frame are projected onto all frame images from the initial frame to the current frame, and the projected coordinates and the coordinates obtained by the tracking process are projected. The three-dimensional motion of the camera and the three-dimensional coordinates of each feature point are calculated by nonlinear minimization.

[0049] By repeating the above processing, the three-dimensional motion of the digital camera in each frame is calculated as a change in the three-dimensional coordinates in the three-dimensional space.

FIG. 8 is an explanatory diagram showing a specific example of the calculation state of the three-dimensional motion. For example, as shown in FIG. 8 (A), the image power captured by the digital camera is changed from the state in which the face of the user 40 as the subject 205 exists in the left area of the captured image, as shown in FIG. 8 (B). User 40 ’s face Assume that the state has changed to exist in the right region of the captured image. In other words, as shown in Fig. 8, as a three-dimensional movement, a rightward movement (positive direction of the X axis) is observed on the image. According to this observation result, the three-dimensional motion calculation unit 12 calculates the parallel movement in the negative direction of the X axis and the amount of the movement.

FIG. 9 is an explanatory diagram showing another specific example of the calculation state of the three-dimensional motion. For example, as shown in FIG. 9B, the image taken by the digital camera is as shown in FIG. 9B from the state where the face of the user 40 as the subject 205 exists near the center of the shot image as shown in FIG. 9A. In addition, it is assumed that the face of the user 40 is changed to a state in which the face of the photographed image is enlarged vertically and horizontally. In other words, as a three-dimensional motion, as shown in FIG. 9, it is assumed that a radial motion due to moving in the direction facing the subject 205 (the negative direction of the z-axis) is observed on the image. According to this observation result, the three-dimensional motion calculation unit 12 calculates the parallel movement in the positive direction of the z axis and the amount of the movement.

[0052] Next, the operation information generation unit 13 uses the image viewer unit (the converter that performs control according to the display control unit 14) to the 3D motion amount (movement direction and movement amount) calculated by the 3D motion calculation unit 12. The operation information is converted into operation information necessary for controlling the display unit 21. If the operation information enabling switch 15 is pressed (on state), the operation information is input to the image viewer unit.

[0053] Here, the movement amount that is the same as the movement amount of the digital camera in the x and y axis directions is set as the scroll amount in the image viewer portion, and the movement amount that is the same as the movement amount in the z axis direction is enlarged or reduced in the image viewer portion. Map to a small scale.

[0054] Note that if the image is greatly scrolled on the viewer, the user needs to greatly translate the terminal even in the real world. This makes scrolling difficult when a large space cannot be secured in the user's environment. In such a case, move the digital camera slightly in parallel while pressing down the operation information enable switch 15, then release the operation information enable switch 15 (turn it off) and move the digital camera to its original position. The desired scroll amount can be obtained by repeating the operation of returning to a plurality of times.

[0055] Further, the movement amount proportional to the movement amount of the digital camera with the same movement amount as the movement amount of the digital camera may be used as the scroll amount or enlargement / reduction scale in the image viewer section. Then, the image viewer unit displays the display image generated based on the input operation information on the display unit 21 and presents it to the user.

FIG. 10 is an explanatory diagram showing a specific example of an image displayed on the display unit 21 by display control of the image viewer unit. Here, the case where the image taken by the digital camera changes from FIG. 8A to FIG. 8B will be described as an example.

For example, assume that an image as shown in FIG. 10C is displayed on the display unit 21 when the image of FIG. 8A is taken by the digital camera. An image shown in FIG. 10C is a partial image 302 of the entire image shown in FIG. FIG. 10A shows an entire image that can be displayed on the display unit 21 by the image viewer unit in this example.

[0059] Next, when the digital camera is moved in the negative direction of the X axis while the operation information validation switch 15 is pressed by the user, the subject 205 force S moves in the positive direction of the _X axis on the captured image of the digital camera. And the image shown in Fig. 8 (B) is captured.

[0060] The operation information generation unit 13 displays the three-dimensional momentum calculated by the three-dimensional motion calculation unit 12 (the momentum of the subject from the state in Fig. 8A to the state in Fig. 8B), and the image viewer unit The operation information is converted into operation information necessary for controlling the display unit 21. Since the operation information enabling switch 15 is pressed, the operation information is input to the image viewer unit.

[0061] When the operation information is input, the image viewer unit moves the frame in which the partial image 302 is located in the movement direction indicated by the operation information (X-axis negative direction) by the movement amount indicated by the operation information. Then, the partial image 301 that is an image surrounded by the moved frame is cut out from the entire image. Then, the image viewer unit changes the display image of the display unit 21 from the partial image 302 shown in FIG. 10 (C) to the partial image 301 shown in FIG. 10 (B).

FIG. 11 is an explanatory diagram showing another specific example of an image displayed on the display unit 15 by display control of the image viewer unit. Here, an example will be described in which the image captured by the digital camera changes from FIG. 9 (A) to FIG. 9 (B).

For example, it is assumed that an image as shown in FIG. 11C is displayed on the display unit 21 when the image of FIG. 9A is taken by the digital camera. The image shown in FIG. 11C is a partial image 303 that is a part of the entire image shown in FIG. Figure 11 (A) shows this. In this example, the entire image can be displayed on the display unit 21 by the image viewer unit.

[0064] Next, when the digital camera is moved in the positive z-axis direction with the operation information enabling switch 15 pressed by the user, the subject 205 appears to move in the negative z-axis direction on the captured image of the digital camera. Thus, the image shown in FIG. 9B is captured.

[0065] The operation information generation unit 13 displays the three-dimensional momentum calculated by the three-dimensional motion calculation unit 12 (the momentum of the subject from the state in Fig. 9A to the state in Fig. 9B), and the image viewer unit The operation information is converted into operation information necessary for controlling the display unit 21. Since the operation information enabling switch 15 is pressed, the operation information is input to the image viewer unit.

[0066] When the operation information is input, the image viewer unit moves the frame indicated by the operation information in the movement direction (z-axis positive direction) indicated by the operation information in the frame where the partial image 303 is positioned. Then, the partial image 301 that is an image surrounded by the moved frame is cut out from the entire image. Then, the image viewer unit changes the display image of the display unit 15 from the partial image 303 shown in FIG. 11 (C) to the partial image 301 shown in FIG. 11 (B).

[0067] In this way, if the operation information enabling switch 15 is in the depressed state, the display screen of the display unit 21 moves as the digital camera moves.

[0068] In the above example, the same processing is performed for the force and other embodiments that are specific examples of the second embodiment. However, when there is a camera subject on the user side as in the second embodiment, and when there is a camera subject on the opposite side of the user as in the first embodiment, the image is captured by the camera. The moving direction of the subject is different from the moving direction of the display image on the display unit 21. That is, when there is a camera subject on the user side, the moving direction of the subject captured by the camera and the moving direction of the display image on the display unit 21 are opposite in the z-axis direction, and in the xy-axis direction. Then the same direction. In addition, when there is a camera subject on the opposite side of the user, the moving direction of the subject imaged by the camera and the moving direction of the display image on the display unit 21 are opposite in the xy axis direction, and the z axis The direction is the same.

[0069] Also, in the above example, when there is a camera subject on the user side, the moving direction of the subject captured by the camera and the moving direction of the display image on the display unit 21 are opposite in the z-axis direction. The force that was supposed to be the same direction in the xy axis direction was reversed in the xy axis direction. The z-axis direction may be the same direction. In this case, the display image on the display unit 21 is moved (scrolled, enlarged Z reduced) in the direction opposite to the direction in which the user moved the digital camera. Similarly, when there is a camera subject on the opposite side of the user, the moving direction of the subject imaged by the camera and the moving direction of the display image on the display unit 21 are opposite in the z-axis direction, It may be the same direction in the xy axis direction

Claims

The scope of the claims

[1] A three-dimensional coordinate input device mounted on a terminal device,

A moving image shooting means for shooting a moving image;

Three-dimensional motion calculation means for calculating a motion in the three-dimensional space of the terminal device as a change in three-dimensional coordinates based on the moving image photographed by the moving image photographing means; and the calculated of the terminal device Based on the exercise, the operation information generating means for generating operation information for changing the image displayed on the display means for displaying the image, and displayed on the display means according to the generated operation information. Display control means for controlling the display means based on predetermined software for changing the image to be obtained.

[2] The apparatus according to claim 1, further comprising an operation information enabling switch for enabling or disabling the operation information generated by the operation information generating means according to a user operation.

[3] The apparatus according to claim 1 or 2, wherein the display unit and the display control unit are controlled by a computer different from a computer that controls the terminal device.

[4] The display unit according to any one of claims 1 to 3, wherein the display unit is installed on a side of the casing of the terminal device opposite to a side on which the moving image shooting unit is installed. Equipment described in.

[5] The display unit according to any one of claims 1 to 3, wherein the display unit is installed on the same side of the casing of the terminal device as the side on which the moving image capturing unit is installed. Equipment described

[6] The moving image photographing means includes a first moving image photographing means and a second moving image photographing means, the display means is installed in the housing, and the first moving image photographing means includes the Installed on the side of the casing where the display means is installed, and the second moving image capturing means is installed on the side of the casing opposite to the side where the display means is installed. The apparatus according to any one of claims 1 to 3.

[7] The display means includes a backlight for brightly displaying a display screen, and the moving image photographing means installed on the same side of the casing together with the display means displays a moving image. The apparatus according to claim 5 or 6, wherein the backlight is used as a light source for illuminating a subject when photographing.

[8] The apparatus according to any one of [1] to [7], further comprising a light source for illuminating a subject when the moving image capturing unit captures a moving image.

[9] The moving image photographing unit is printed with a texture prepared in advance to obtain a moving image for causing the three-dimensional operation calculating unit to calculate the operation of the terminal device in the three-dimensional space. 9. The apparatus according to claim 1, wherein the apparatus photographs a subject.

[10] A three-dimensional coordinate input device mounted on a terminal device,

A moving image shooting means for shooting a moving image;

Three-dimensional motion calculation means for calculating a motion in the three-dimensional space of the terminal device as a change in three-dimensional coordinates based on the moving image photographed by the moving image photographing means; and the calculated of the terminal device Based on the exercise, the operation information generating means for generating operation information for changing the image displayed on the display means for displaying the image, the generated operation information, the display means as a predetermined software Operation information output means for outputting to the display control means to be controlled based on,

A three-dimensional coordinate input device.

[11] A three-dimensional coordinate input method using a terminal device,

A step of photographing a moving image by a moving image photographing means installed in the terminal device, and a step of calculating a motion in the three-dimensional space of the terminal device as a change of three-dimensional coordinates based on the photographed moving image;

A step of generating operation information for changing the image displayed on the display means for displaying the image based on the calculated movement of the terminal device in the three-dimensional space;

Controlling the display means based on predetermined software, and changing an image displayed on the display means according to the generated operation information;

A three-dimensional coordinate input method. A computer program that allows a computer to input 3D coordinates using a terminal device.

A procedure for capturing a moving image by a moving image capturing means installed in the terminal device; a procedure for calculating a motion of the terminal device in a three-dimensional space as a change in three-dimensional coordinates based on the captured moving image; ,

A procedure for controlling the display means based on predetermined software and changing an image displayed on the display means in accordance with the generated operation information;

A computer program comprising: