JP2017021495A - 3D coordinate detection system, information processing apparatus, program, and 3D coordinate detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance a degree of freedom in rotating, moving, and enlarging/reducing a 3D object.SOLUTION: A 3D coordinate detection system includes: an electronic pen having a light-emitting unit, and a button which generates a signal for entering 3D object operation mode which enables 3D object operation when in ON state; and an information processing apparatus comprising a display unit for displaying a 3D object, imaging units arranged at both top ends of the display unit to image the light-emitting unit, a coordinate detection unit which detects 3D coordinates of the electronic pen on the basis of images of the light-emitting unit captured by the imaging units, and a control unit which selects hovering operation mode where the 3D object is directly rotated, moved, or enlarged/reduced along with the motion of the electronic pen, when the button of the electronic pen is ON, in a predetermined area on a front surface of the display unit, or writing operation or gesture operation is enabled in the predetermined area, while the 3D object remains still, when the button is OFF.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a 3D coordinate detection system, an information processing apparatus, a program, and a 3D coordinate detection method.

  An electronic whiteboard and a control method are already known that use an electronic pen to touch and trace a display, display its trajectory on a display surface, and display the content to be displayed (see FIG. 1).

FIG. 1 is a conceptual diagram of an electronic whiteboard.
In FIG. 1, an electronic whiteboard connected to a computer and having a pair of cameras disposed on both ends of the upper surface is shown, and an electronic pen is displayed on the display.

  In recent years, with the spread of computers having advanced graphics functions, contents using 3DCG (3 Dimensional Computer Graphics) have become widespread. However, the operation of 3D objects with electronic pens is limited.

FIG. 2 is a conceptual diagram showing the relationship between the display and the electronic pen.
As shown in FIG. 2, since the electronic pen can only move in a plane in contact with the display, there is a problem that operations such as movement and rotation with depth are insufficient.
A 3D object (3-dimension object) is data describing a three-dimensional object.
Various proposals have been made to solve this type of problem (see, for example, Patent Document 1).
Patent Document 1 discloses that 3D objects are operated for the purpose of operation, and pen pressure information is operated as depth coordinates.

  However, the invention described in Patent Document 1 has a problem that a method of operating a 3D object using pen pressure or inclination has a low degree of freedom, for example, it cannot be freely rotated.

  Accordingly, an object of the present invention is to improve the degree of freedom of rotation, movement, and enlargement / reduction of a 3D object.

  In order to solve the above-mentioned problem, the invention described in claim 1 is directed to an electronic pen having a light emitting unit and a button for generating a signal for entering a 3D object operation mode in which a 3D object can be operated when turned on. Display unit for displaying an object, photographing unit arranged at both upper ends of the display unit, photographing unit for photographing the light emitting unit, coordinate detection for detecting 3D coordinates of the electronic pen based on images of the light emitting unit from both photographing units When the button of the electronic pen is on in a predetermined area on the front surface of the display unit and the display unit, the 3D object is directly rotated, moved, and scaled according to the movement of the electronic pen, and the button is turned off. In this case, the 3D object has a control unit that switches to a hovering operation mode in which the 3D object is in a stationary state and can perform a writing operation or a gesture operation in the predetermined area. A processing unit, characterized by comprising a.

  According to the present invention, the degree of freedom of rotation, movement, and enlargement / reduction of a 3D object can be improved.

It is a conceptual diagram of an electronic whiteboard. It is a conceptual diagram which shows the relationship between a display and an electronic pen. It is an example of the hardware block diagram of a 3D coordinate detection system. 3 is a conceptual diagram of a 3D object 105 in a virtual operation space of an electronic pen. FIG. FIG. 3 is a conceptual diagram showing a relationship between a virtual operation space of an electronic pen and a virtual operation space expanded by writing pressure information and a 3D object 105. It is explanatory drawing of a hovering state. 1 is a conceptual diagram of an information processing apparatus and an electronic pen according to an embodiment. It is a figure which shows the state which is 3D-operating with the electronic pen shown in FIG. It is an example of the flowchart which shows the relationship of three scanning modes. FIG. 4 is a diagram showing a state where a button 104 of the electronic pen 103 is turned off from on. It is a figure which shows the state which is performing writing operation with the electronic pen 103 in 3D operation mode. It is an example of the functional block diagram of the 3D coordinate detection system shown in FIG. It is explanatory drawing at the time of moving the electronic pen 103 to a X, Y, Z direction. 4A is an example of a flowchart when the electronic pen 103 shown in FIG. 4 is linearly moved, FIG. 4B is a reduced view of FIG. 4, and FIG. It is explanatory drawing. (A)-(d) is explanatory drawing of rotation operation by the electronic pen 103. FIG. (A) is an example of a flowchart of the rotation operation, and (b) is an example of a movement locus of the electronic pen 103. (A) shows the rotation angle when a circular arc is drawn at the tip of the electronic pen 103, and (b) shows the state before and after the rotation of the 3D object 105. FIG. 6 is an explanatory diagram of an enlargement / reduction operation by the electronic pen 103. (A) is an example of a flowchart of an enlargement / reduction operation, (b) is an explanatory view of an enlargement / reduction scale, and (c) is an explanatory view of an enlargement / reduction operation instruction arrow. (A) is an example of a flowchart for switching the linear movement operation, the rotation operation, or the enlargement / reduction operation of the 3D object 105, (b) is an example of an arc-shaped movement locus of the electronic pen 103, and (c () Is an example of a linear movement locus of the electronic pen 103, and (d) is an example of a zigzag movement locus of the electronic pen 103.

<Overview>
An embodiment of the present invention will be described. Specifically, when operating the 3D object 105, it has the following characteristics.
In short, the present invention is characterized in that it can be operated with a high degree of freedom by natural movement with respect to the optical system of (camera + light-emitting electronic pen).

<Embodiment>
The above features will be specifically described with reference to the following drawings.
[Constitution]
FIG. 3 is an example of a hardware block diagram of the 3D coordinate detection system.

The 3D coordinate detection system includes an information processing apparatus 100 and an electronic pen 103.
The information processing apparatus 100 includes a CPU 11, a ROM 12, a RAM 13, an SSD 14, a network controller 15, an external storage controller 16, a camera controller 120, a GPU 112, and a capture device 111 that are electrically connected via a bus line 118.

  A CPU (Central Processing Unit) 11 executes application software (hereinafter referred to as an application) to control the entire operation of the coordinate detection system. A ROM (Read Only Memory) 12 stores an IPL (Initial Program Loader) and the like, and mainly stores a program executed by the CPU 11 at startup. A RAM (Random Access Memory) 13 is a work area when the CPU 11 executes an application. An SSD (Solid State Drive) 14 is a nonvolatile memory in which an application 119 for a coordinate detection system that detects 3D coordinates and various data are stored. A control unit is configured by the CPU 11, the ROM 12, and the RAM 13.

  The network controller 15 performs processing based on a communication protocol when communicating with a server or the like via a network (not shown). The network is a LAN (Local Area Network) or a WAN (Wide Area Network: for example, the Internet) to which a plurality of LANs are connected.

  The external storage controller 16 performs writing to or reading from the removable external memory 117. The external memory 117 is a storage medium such as a USB memory or an SD card. The capture device 111 captures (captures) video displayed on the display 101 as a display unit by the PC 110. A GPU (Graphics Processing Unit) 112 is a drawing-dedicated processor that calculates the pixel value of each pixel of the display 101. The display controller 113 outputs the image created by the GPU 112 to the display 101.

  The camera controller 120 is connected to two cameras 102 that are arranged at both upper ends of the display unit and serve as photographing units for photographing the electronic pen 103. For example, coordinates are detected by a triangulation method. Details will be described later.

  In the present embodiment, the information processing apparatus 100 does not need to communicate with the electronic pen 103 as an instruction unit, but may have a communication function. In this case, as illustrated, the information processing apparatus 100 includes an electronic pen controller 116 and receives a contact signal from the electronic pen 103 (when the electronic pen 103 includes a notification unit). Thus, the information processing apparatus 100 can detect whether or not the tip is in contact with the display 101.

  The application for the coordinate detection system may be distributed while being stored in the external memory 117, or may be downloaded from a server (not shown) via the network controller 15. The application may be compressed or executable.

FIG. 4 is a conceptual diagram of the 3D object 105 in the virtual operation space as a predetermined area on the front surface of the display 101 of the electronic pen 103.
FIG. 6 is an explanatory diagram of a hovering state.
The optical display using the camera 102 and the electronic pen 103 can also detect the hovering coordinates shown in FIG. The hovering is a state in which the electronic pen 103 does not directly contact the display 101, the spatial coordinates of the electronic pen 103 can be detected, and the planar hovering coordinates corresponding to the display 101 can also be calculated. Further, by hovering, the 3D object 105 can be written and gestured while still. As a result, the existing optical handwriting system (electronic pen + camera) can detect the spatial coordinates of the electronic pen 103 in the non-contact state, so that the 3D object 105 can be operated in the non-contact state. The 3D object 105 displayed on the display 101 can be moved and rotated with a high degree of freedom by natural movement in human space.

FIG. 5 is a conceptual diagram showing the relationship between the virtual operation space of the electronic pen 103 and the virtual operation space expanded by the pen pressure information and the 3D object.
The difference from the virtual operation space shown in FIG. 5 is that a virtual space is created behind (in the back) of the display 101 based on writing pressure information by the electronic pen 103.
Even in such a case, the 3D object can be moved, rotated, reduced and enlarged.

FIG. 7 is a conceptual diagram of the information processing apparatus and the electronic pen 103 according to an embodiment. FIG. 8 is a diagram illustrating a state in which 3D operation is performed with the electronic pen 103 illustrated in FIG. 7.
Specifically, reference numeral 101 denotes a display, which is a flat device capable of displaying a handwritten locus and moving the displayed content.
Reference numeral 102 denotes a camera. In order to detect the coordinates of the electronic pen 103, at least two cameras 102 need to be installed. The position of the electronic pen 103 is detected by triangulation. Furthermore, the camera 102 can detect hovering coordinates in the case of non-contact as shown in FIG. 6 by installing a position slightly higher than the screen of the display 101.

  An electronic pen 103 shown in FIG. 8 is a pen-type indicating member having a light emitting point 103a as an infrared point light source, for example. The position of the light emitting point 103a is detected from the image of the camera 102 by an image processing method, and the coordinates in the space of the electronic pen 103 can be calculated. The effect of using the infrared point light source is that it is not visible to the user and is not bothered during operation.

  The button 104 is a button that generates a signal requesting that a 3D object operation mode in which a 3D object can be rotated, moved, and enlarged / reduced when turned on by a user. By providing the button 104 on the electronic pen 103, it is possible to switch between a writing operation and a gesture operation, which are 2D operations, and a movement operation, a rotation operation, or a reduction / enlargement operation, which are 3D operations. The off state is a state where the button 104 is not pressed, and the on state is a state where the button 104 is pressed.

  In the case of 3D support, the content displayed on the display 101 treats the calculated distance between the electronic pen 103 and the display 101 as the 3D depth coordinate and the target 3D object (hereinafter referred to as 3D object). ) Operate 105. When 3D is not supported, the state is the same as when the button 104 is off.

  The 3D object 105 displays the 3D object 105 on the display 101. This display can be realized by existing webGL (standard specification for displaying 3D computer graphics in a web browser), CAD (Computer Aided Design), or the like. It can also be developed with 3D-compatible libraries such as OpenGL (Open Graphics Library).

  There are three operation modes according to the contact state between the electronic pen 103 and the display 101 and the state of the button 104.

<Operation 1>
FIG. 9 is an example of a flowchart showing the relationship between the three scanning modes.
First, it is determined whether or not the electronic pen 103 is in contact with the display 101 (step S201). When the electronic pen 103 is in contact with the display 101 (step S201 / Y), the writing operation mode is set (step S203). ).

  Here, the writing operation mode is a mode in which the electronic pen 103 and the display 101 are in contact with each other when the button 104 is off, and drawing and gesture operations can be performed like a normal touch panel.

  When the electronic pen 103 is not in contact with the display 101 (step S201 / N), it is determined whether or not the button 104 is pressed (step S202), and when the button 104 is not pressed (step S202 / N). Then, the hovering operation mode is set (step S204).

  Here, the hovering operation mode is that the button 104 is off and the electronic pen 103 can detect the spatial coordinates of the electronic pen 103 without directly touching the display 101 (non-contact state) as shown in FIG. In this mode, coordinates can be displayed on the display 101.

Here, the hovering state will be described with reference to FIG.
A pair of cameras 102 are disposed at both ends of the upper surface of the display 101 shown in FIG. The intersection of the line connecting the camera 102 and the light emitting point 103a of the electronic pen 103 and the image plane is shown on the image plane. A broken line from the camera 102 toward the display 101 is a line parallel to the display 101 and toward the electronic pen 103. That is, the straight line from the camera 102 toward the light emitting point 103a of the electronic pen 103 is downward with respect to the broken line, indicating that the camera 102 is in a position floating with respect to the display 101.
An electronic pen 103 is arranged on the display 101 in a hovering state. On the display 101, spatial coordinates (X, Y, Z) of the world coordinate system are shown.

  Here, the world coordinate system is a coordinate system representing the entire space mainly used in the field of 3DCG. A coordinate system for indicating the position of an object in space, and is used to handle display and movement of the object. Sometimes called the “global coordinate system”. On the other hand, a coordinate system used to handle the shape and deformation of each object is called a local coordinate system or a body coordinate system, and the center of gravity of each object is the origin.

  Both cameras 102 are arranged at a predetermined distance in the Z direction with respect to the XY plane of the display 101. The coordinates of the light emitting point 103a at the tip of the electronic pen 103 are set as the spatial coordinates of the electronic pen 103 (X, Y, Z). The spatial coordinates (X, Y, Z) of the electronic pen 103 can be detected by triangulation. The hovering coordinate is (X, Y).

  Returning to FIG. 9, when the button 104 is pressed (step S202 / Y), the 3D object operation mode is set (step S205).

  Here, the operation mode of the 3D object 105 is a mode for operating the 3D object displayed on the display 101 while the electronic pen 103 is not in contact with the display 101 and the button 104 is turned on. The 3D object operation mode is the main feature of the present application.

  The method of selecting the target 3D object is the spatial coordinates (X, Y, Z) when the button 104 of the electronic pen 103 is pressed. The plane coordinate of the corresponding display 101 is (X, Y). The 3D object 105 corresponding to this coordinate is an operation target. Then, the object operation is performed while the button 104 is pressed.

The 3D operation of the electronic pen 103 can be performed in the virtual operation space of the electronic pen 103 in front of the display 101. As shown in FIG. 8, the operation space of the electronic pen 103, that is, a range in which the position of the electronic pen 103 can be detected. By pressing the button 104 of the electronic pen 103, the 3D object 105 is selected, and this position is set as the initial position of the pen operation.
The operation on the 3D object 105 displayed on the display 101 is performed on the 3D object (the 3D object does not actually exist, but was created because it is easy to understand).
When operating the electronic pen 103, the camera 102 detects the position and movement of the electronic pen 103, and the 3D object 105 displayed on the display 101 updates the position according to the operation.

Furthermore, since the moving range of the electronic pen 103 is large and the operator can move naturally, the moving distance of the electronic pen 103 is measured as the moving distance of the 3D object 105 as shown in the following formula (1). .
Movement distance d of 3D object 105 = k × movement distance D of electronic pen 103 (1)
(However, 0 ≦ k ≦ 1)

  There are mainly three 3D operations of the electronic pen 103. That is, a linear movement operation, a rotation operation, and an enlargement / reduction (scaling) operation. A 3D object 105 indicated by a solid line in FIG. 8 is shown as a 3D object indicated by a broken line by an enlargement operation of the electronic pen 103. In the operation space of the electronic pen 103, the button 104 is pressed at the tip of the pen operation initial position at the initial position of the electronic pen 103, and dragged to the position shown in the figure to be enlarged.

FIG. 10 is a diagram illustrating a state where the button 104 of the electronic pen 103 is turned off from on.
By turning the button 104 from on to off, the electronic pen 103 is switched from 3D operation to 2D operation, and the 3D object 105 cannot be moved, rotated, reduced or enlarged, but writing operation and gesture operation are possible. .

  FIG. 11 is a diagram illustrating a state in which the electronic pen 103 is performing a writing operation in the 3D operation mode. By moving the electronic pen 103 on the display 101, a circle, an arrow, and a broken line can be displayed.

FIG. 12 is an example of a functional block diagram of the 3D coordinate detection system shown in FIG.
The information processing apparatus 100 includes a 3D object operation module 305, an image processing module 301, a coordinate calculation module 302 as a coordinate detection unit, a writing operation module 303, and other applications 304.
The 3D object operation module 305 includes a movement operation module 306, an enlargement / reduction operation module 307, and a rotation operation module 308.

The image processing module 301 can analyze an image captured by the camera 102, cut out a target object (for example, the electronic pen 103) from the background, and obtain shape information and image coordinate information.
The coordinate calculation module 302 calculates world coordinates from image coordinates. As shown in FIG. 5, the image coordinates are coordinate positions of the electronic pen 103 projected onto the image plane. In world coordinates, it is the coordinate position of the electronic pen 103 in space.
The writing operation module 303 can draw a handwritten trajectory on the display 101 or make an object to be operated react based on the calculation result of the world coordinate module.
The 3D object operation module 305 is a module that performs operations in the 3D object operation mode. Furthermore, three sub-modules including a movement operator module 306, a rotation operator module 308, and an enlargement / reduction operator module 307 are included according to a linear movement operation, a rotation operation, or an enlargement / reduction operation.

Hereinafter, a specific 3D object operation method will be described with reference to FIG.
FIG. 13 is an explanatory diagram when the electronic pen 103 is moved in the X, Y, and Z directions.

As shown in FIG. 13, the target 3D object 105 is selected by pressing the button 104 of the electronic pen 103, and this position is set as the initial position of the pen operation. When the electronic pen 103 is moved while the button 104 is pressed, the position of the 3D object 105 displayed on the display 101 is also moved in the same direction. The movement of the electronic pen 103 has three degrees of freedom. That is, it can move in any direction of X direction, Y direction, Z direction, or XYZ.

<Operation 2>
14A is an example of a flowchart when the electronic pen 103 shown in FIG. 4 is linearly moved, FIG. 14B is a reduced view of FIG. 4, and FIG. 6 is an explanatory diagram of a moving distance of a pen 103. FIG.

1. Linear movement operation In step S501 in the flowchart, a 3D object is selected. The target 3D object 105 is selected by pressing the button 104 of the electronic pen 103, and this position is set as the initial position of the pen operation. In the upper right of FIG. 14, a state in which the 3D object moves on the display 101 as the electronic pen 103 moves is displayed.

In step S502, the world coordinates of the electronic pen 103 are calculated. That is, the world coordinates (x0, y0, z0) of the electronic pen 103 at that time are calculated.
In step S503, the world coordinates (x0, y0, z0) are updated to obtain new world coordinates (xt, yt, zt). After the time Δt has elapsed, the world coordinates of the electronic pen 103 at a new point in time are recalculated.
The world coordinates before update (x0, y0, z0) and the world coordinates after update (xt, yt, zt) are displayed in the right center of FIG.

  In step S504, the moving distance of the electronic pen 103 is calculated. As shown in FIG. 14C, the movement distance (Dx, Dy, Dz) in each direction of the electronic pen 103 is calculated from the world coordinates calculated in steps S502 and S503.

In step S505, the moving distance of the 3D object 105 is converted. That is, the movement distance (dx, dy, dz) of each direction of the 3D object 105 is calculated by the mathematical formula (1). Save in the form of the translation matrix of equation (2).
In step S506, the 3D object 105 is moved (redrawn). The coordinates after movement are calculated, and the 3D object 105 is redrawn at the new coordinate position. For each point of the 3D object 105, the coordinates after movement = the translation matrix * the coordinates before movement.
Since there is a minute vibration in the movement of a human hand, a small vibration can be suppressed by calculating the moving direction of how many frames and setting the average direction as the moving direction of the 3D object 105.

<Operation 3>
2. Rotation Operation The target 3D object 105 is selected by pressing the button 104 of the electronic pen 103, and this position is set as the initial position of the operation of the electronic pen 103. The 3D object 105 can be rotated in that direction by gradually drawing a circumference or arc in a certain direction while keeping the button 104 pressed.

FIGS. 15A to 15D are explanatory diagrams of the rotation operation by the electronic pen 103. FIG.
As shown in FIG. 15A, the rotation operation is basically a rotation operation with respect to each axis in the XYZ directions. That is, as shown in FIG. 15B, when the light emitting point 103a at the tip of the electronic pen 103 is rotated in the direction of the dashed arrow with the X direction axis as the rotation axis, the 3D object 105 rotates on the X direction axis. As shown in FIG. 15C, when the light emitting point 103a, which is the tip of the electronic pen 103, is rotated in the direction of the broken line arrow with the Y direction axis as the rotation axis, the 3D object 105 rotates on the Y direction axis. Similarly, as shown in FIG. 15D, when the light emitting point 103a at the tip of the electronic pen 103 is rotated in the direction of the broken line arrow as shown in FIG. Rotate with.

FIG. 16A is an example of a flowchart of the rotation operation, and FIG. 16B is an example of the movement trajectory of the electronic pen 103.
In step S601 of FIG. 16A, a 3D object is selected. The target 3D object 105 is selected by pressing the button 104 of the electronic pen 103, and this position is set as the initial position of the pen operation.
In step S602, the world coordinates of the electronic pen 103 are calculated. The world coordinates (x0, y0, z0) of the electronic pen 103 at that time are calculated.

In step S603, the world coordinates are updated. After a certain time interval NΔt has elapsed, the coordinates of each position during NΔt are calculated, and the movement locus of the electronic pen 103 as shown in FIG. 16 is recorded.
In step S604, the movement trajectory of the electronic pen 103 is analyzed to estimate the rotation direction. Specifically, the direction of rotation is estimated by analyzing the trajectory of the electronic pen 103 (such as straight line fitting).
In step S605, the rotation angle is calculated.

17A shows a rotation angle when an arc is drawn at the tip of the electronic pen 103, and FIG. 17B shows a state before and after the rotation of the 3D object 105. As shown in FIG.
As shown in FIGS. 17A and 17B, the electronic pen 103 is an example that rotates around the Z-direction axis by θ degrees. The rotation angle θ of the 3D object 105 corresponds to the rotation angle of the electronic pen 103.
In step S606, the 3D object 105 is rotated, that is, redrawn. The coordinates after rotation are calculated, and the 3D object 105 is redrawn at the new coordinate position. Each point of the 3D object 105 is expressed by Equation (2).

          Coordinate after rotation = rotation matrix * coordinate before rotation ...... (3)

  Examples of the rotation matrix include a 4 × 4 rotation matrix (X axis: Formula (4)), a rotation matrix (Y axis: Formula (5)), and a rotation matrix (Z axis: Formula (6)).

<Operation 4>
3. Enlarging / reducing operation By this enlarging / reducing operation, only the size can be changed without changing the shape of the 3D object 105.
FIG. 18 is an explanatory diagram of the enlargement / reduction operation by the electronic pen 103.
In order to distinguish between the enlargement / reduction operation and the movement operation, instruction arrows for the enlargement / reduction operation are displayed at the corners of the 3D objects 105a and 105b. The size of the 3D object 105 can be changed by selecting this arrow and gradually moving in the enlargement direction and the reduction direction.

FIG. 19A is an example of a flowchart of an enlargement / reduction operation, FIG. 19B is an explanatory view of an enlargement / reduction scale, and FIG. 19C is an explanatory view of an enlargement / reduction operation instruction arrow. .
In step S701, a 3D object is selected. While confirming the coordinate position in the hovering state, the user can select the enlargement / reduction mode of the target 3D object by pressing the button 104 of the electronic pen 103 within the range of the enlargement / reduction instruction arrow.
As shown in FIG. 19 (c), the pointing arrow is written at a cubic corner surrounding the 3D object.

In step S702, the world coordinates of the electronic pen 103 are calculated. The world coordinates (x0, y0, z0) of the electronic pen 103 at that time are calculated.
In step S703, the world coordinates are updated. After a time interval Δt has elapsed, new world coordinates (xt, yt, zt) are recalculated. As shown in FIG. 19B, the movement distances (Dx, Dy, Dz) in the x, y, and z directions can also be calculated.
In step S704, the movement distance (Dx, Dy, Dz) calculated in step S703 is converted into the movement distance (dx, dy, dz) of the 3D object 105b by the above-described equation (1).
In step S705, an enlargement / reduction scale in the x, y, and z directions is calculated from the movement distance (dx, dy, dz). Specifically, as shown in FIG. 19C, the length of the cubic x, y, z surrounding the 3D object 105b is (Lx, Ly, Lz). The scaling scale (sx, sy, sz) is expressed by the following mathematical formula (7).
(sx, sy, sz) = (dx / Lx, dy / Ly, dz / Lz) (7)

In step S706, enlargement / reduction (redrawing), which is an update of the 3D object 105b, is performed. Calculate the scaled coordinates and redraw the 3D object at the new coordinates. Redraw each point of the 3D object 105b using the following formula (8).
Scaled coordinates = Scaled matrix * Coordinates before scaling ... (8)
The enlargement / reduction matrix is expressed by Equation (9).

As for the switching method of the above-described linear movement operation, rotation operation, and enlargement / reduction operation of the 3D objects 105, 105a, and 105b, in normal 3D software, one operation is often selected from a menu.
Therefore, in this application, the operation is automatically determined so as not to take extra time. A specific determination method is shown in FIGS.

  FIG. 20A is an example of a flowchart for switching between a linear movement operation, a rotation operation, and an enlargement / reduction operation of a 3D object, and FIG. 20B is an example of an arc-shaped movement locus of the electronic pen 103. . FIG. 20C is an example of a linear movement locus of the electronic pen 103, and FIG. 20D is an example of a zigzag movement locus of the electronic pen 103.

In step S1701 of the flowchart shown in FIG. 20A, it is determined that the operation mode is the 3D object operation mode as shown in FIG. 9, and the 3D object operation is started.
In step S1702, the world coordinates of the electronic pen 103 at the start time (when the button 104 is pressed) are calculated.
In step S1703, it is determined whether or not the world coordinates fall within the action range of the instruction arrow that instructs the enlargement / reduction operation.

In step S1704, if the determination result in step S1703 is “within the action range of the pointing arrow”, the operation from now on is determined to be an enlargement / reduction operation.
In step S1705, if the determination result in step S1703 is “outside the action range of the pointing arrow”, further operation analysis is performed. First, as shown in FIG. 20B, the movement locus of the electronic pen 103 is recorded within a certain time interval NΔt, and the world coordinates of each point are calculated.
In step S1706, the movement locus of the electronic pen 103 is fitted with a straight line. Specifically, assuming that a straight line is y = ax + b, the straight line parameter a and the parameter b are calculated by the least square method.

In step S1707, an error error between the recorded movement locus of the electronic pen 103 and the calculated straight line is calculated. Calculate by setting a certain threshold th.
In step S1708, if the error calculated in step S1707 <th, it is considered that the movement trajectory of the electronic pen 103 can be fitted with a straight line, so it is determined that the movement is a straight line (FIG. 20C).

  In step S1709, when the error calculated in step S1707> th, the locus error is considerably away from the straight line as shown in FIG. For this reason, it is considered that the movement locus of the electronic pen 103 cannot be fitted with a straight line. In the case of the present application, it is determined as a rotation operation.

  Although the movement of a human hand in the air may be unstable, the instability can be reduced by operating with a large movement while avoiding a minute movement according to Equation (1). Further, as described above, it is considered that the instability caused by estimating the motion using the average of how many frames can be reduced.

<Effect>
As described above, since the coordinates, writing, and gesture of the hovering state can be detected by the optical display using the camera and the electronic pen, the degree of freedom of rotation, movement, and enlargement / reduction of the 3D object can be improved.

  It should be noted that the system configuration in which the electronic pen and the information processing apparatus described in this embodiment are connected is an example, and it goes without saying that there are various system configuration examples depending on applications and purposes.

<Program>
The information processing apparatus according to the present invention described above is realized by a program that causes a computer to execute processing. As an example, the case where the function of the present invention is realized by a program will be described below.

For example,
A computer-readable program of an information processing apparatus having a display unit, a photographing unit, a coordinate detection unit, and a control unit,
On the computer,
A computer-readable program of an information processing apparatus having a display unit, a photographing unit, a coordinate detection unit, and a control unit,
On the computer,
The procedure that the display unit displays 3D objects,
Procedures for photographing the light emitting part of the electronic pen placed on the upper ends of the display part,
A procedure for the coordinate detection unit to detect the 3D coordinates of the electronic pen based on the image of the light emitting unit from both imaging units,
When the button of the electronic pen is on in the specified area on the front of the display unit, the 3D object is directly rotated, moved and scaled according to the movement of the electronic pen. When the button is off, the 3D object is stationary. The procedure for switching to the hovering operation mode that allows writing and gesture operations within a specified area
A program for executing

  Such a program may be stored in a computer-readable storage medium.

<Storage medium>
Here, examples of the storage medium include computer-readable storage media such as CD-ROM, flexible disk (FD), and CD-R, semiconductor memories such as flash memory, RAM, ROM, and FeRAM, and HDD.

  CD-ROM is an abbreviation for Compact Disc Read Only Memory. Flexible disk means Flexible Disk (FD). CD-R is an abbreviation for CD Recordable. FeRAM is an abbreviation for Ferroelectric RAM and means a ferroelectric memory. HDD is an abbreviation for Hard Disc Drive.

  The above-described embodiment shows an example of a preferred embodiment of the present invention, and the present invention is not limited thereto, and various modifications can be made without departing from the scope of the invention. is there. For example, in the above-described embodiment, the case where an infrared point light source is used for the light emitting unit has been described. However, the present invention is not limited to this, and a visible light point light source may be used.

DESCRIPTION OF SYMBOLS 100 Information processing apparatus 101 Display 102 Camera 103 Electronic pen 103a Light emission point 104 Button 105, 105a, 105b 3D object 110 PC
116 Electronic Pen Controller 117 External Memory 118 Bus Line 119 Application 120 Camera Controller 301 Image Processing Module 302 Coordinate Calculation Module 303 Writing Operation Module 304 Other Application 305 3D Object Operation Module 306 Movement Operator Module 307 Enlargement / Reduction Operator Module 308 Rotation Operator module

JP 2007-536652 A

Claims (6)

An electronic pen having a light emitting unit and a button for generating a signal for entering a 3D object operation mode capable of operating a 3D object when turned on;
3D coordinates of the electronic pen are detected based on a display unit that displays the 3D object, an imaging unit that is disposed at both upper ends of the display unit, and that captures the light emitting unit, and images of the light emitting unit from both imaging units. When the button of the electronic pen is on in a predetermined area on the front surface of the coordinate detection unit and the display unit, the 3D object is directly rotated, moved, and enlarged / reduced according to the movement of the electronic pen, and the button An information processing apparatus having a control unit that switches to a hovering operation mode in which the 3D object remains stationary and can perform a writing operation and a gesture operation in the predetermined area when is off;
A 3D coordinate detection system comprising:   The 3D coordinate detection system according to claim 1, wherein when the electronic pen is in contact with the display unit, a writing operation mode in which a writing operation is possible is set.   The 3D coordinate detection system according to claim 1, wherein the light emitting unit of the electronic pen emits infrared light. A display unit for displaying a 3D object;
An imaging unit that is arranged at both upper ends of the display unit and shoots the light emitting unit of the electronic pen;
A coordinate detection unit that detects 3D coordinates of the electronic pen based on images of the light emitting unit from the two imaging units;
When the button of the electronic pen is on in a predetermined area on the front surface of the display unit, the 3D object is directly rotated, moved, and enlarged / reduced according to the movement of the electronic pen, and when the button is off, the 3D object is operated. An information processing apparatus comprising: a control unit that switches to a hovering operation mode in which a writing operation and a gesture operation can be performed in the predetermined area while still. A computer-readable program of an information processing apparatus having a display unit, a photographing unit, a coordinate detection unit, and a control unit,
In the computer,
A procedure in which the display unit displays a 3D object;
A procedure for photographing the light emitting part of the electronic pen, which is disposed at both upper ends of the display part,
A procedure in which the coordinate detection unit detects 3D coordinates of the electronic pen based on images of the light emitting unit from the two imaging units;
When the button of the electronic pen is on in a predetermined area on the front surface of the display unit, the control unit directly rotates, moves, and enlarges / reduces the 3D object according to the movement of the electronic pen, and the button A procedure for switching to a hovering operation mode in which the 3D object is in a stationary state and can perform a writing operation and a gesture operation in the predetermined area when off;
A program for running A step of preparing an electronic pen having a light emitting unit and a button that is in a 3D object operation mode in which a 3D object can be operated when turned on;
Photographing the light emitting part of the electronic pen with photographing parts arranged at both upper ends of the display part for displaying the 3D object;
Detecting 3D coordinates of the electronic pen based on images of the light emitting unit from the two imaging units;
When the button of the electronic pen is turned on within a predetermined area on the front surface of the display unit, the 3D object is directly rotated, moved, and scaled according to the movement of the electronic pen, and when the button is off, the 3D object is operated. A step of switching to a hovering operation mode in which an object can perform a writing operation and a gesture operation in the predetermined area while being in a stationary state;
A 3D coordinate detection method comprising: