JP2009176138A - Projection user interface system and mobile robot therewith - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a projection user interface system that allows operations like mouse clicks with ease and at low cost. <P>SOLUTION: The projection user interface system includes: a projector 103 for projecting a conversation window; a camera 104 for imaging the conversation window projected by the projector 103; a bright point detection unit 108 for detecting the movement of the gravitational center of a bright point created by a pointer on the conversation window from the image of the conversation window captured by the camera 104; and a bright point fluctuation detecting unit 110 for detecting the fluctuation of the gravitational center of the bright spot. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a projection type user interface system and a mobile robot equipped with the same.

  There is a high need to directly perform operations similar to mouse operations on a personal computer using a laser pointer or the like on a screen projected by a projector or the like. As disclosed in Patent Documents 1 to 5, the position of a bright spot can be measured by superimposing a bright spot from a pointer on an image projected by a projector and photographing this with a camera. That is, data corresponding to the position of the mouse cursor on the personal computer can be calculated.

  More specifically, a projector and a camera are combined, and a calibration image is captured to associate the imaging region with the region on the camera imaging screen. Then, if a bright spot is given by a pointer that emits light, the bright spot coordinates on the screen can be obtained by camera imaging. If this bright spot coordinate is used as an input of a mouse cursor, the cursor of the computer can be moved by the pointer.

By the way, when an actuator is provided like a robot, the projected image is not necessarily projected onto the white screen for projection. Therefore, a user interface that can be projected onto various objects in a real environment is required.
JP-A-8-331667 JP 2003-36142 A JP-A-8-123612 JP 2000-276297 A JP-A-10-91344

  In the above technique, although the cursor can be moved by the pointer, an operation corresponding to a mouse click (hereinafter referred to as a mouse click-like operation) cannot be performed. Therefore, for example, a method is conceivable in which a switch is provided on the pointer and a command corresponding to a mouse click is transmitted to the computer by wireless communication. However, the mechanism is complicated and expensive.

  The techniques disclosed in Patent Documents 1 to 5 are all user interfaces projected onto a screen, and are not considered for user interfaces that can be projected onto various objects in a real environment.

  The present invention has been made in view of the above, and an object of the present invention is to provide a projection type user interface system capable of a mouse click-like operation easily and at low cost.

The first aspect of the present invention is:
A projector that projects a dialogue window;
A camera that captures the dialogue window projected by the projector;
A bright spot detection unit that detects a movement of the center of gravity of a bright spot by a pointer on the dialog window from an image of the dialog window captured by the camera;
The projection type user interface system includes a bright spot fluctuation detecting unit that detects fluctuation of the center of gravity of the bright spot.

  According to the present invention, it is possible to provide a projection type user interface system capable of a mouse click-like operation easily and at low cost.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted as necessary for the sake of clarity.

  FIG. 1 shows the configuration of a projection type user interface system according to the embodiment. This projection type user interface system is particularly suitable for a mobile robot. As shown in FIG. 1, the projection type user interface system according to the embodiment includes a controller 101, a dialog window generation unit 102, a projector (projector) 103, a camera 104, an image comparison unit 106, an area calibration unit 107, and a brightness check. A light emitting unit 108, a bright spot stationary timer 109, and a bright spot fluctuation detecting unit 110 are provided.

  The projection type user interface system according to the embodiment is optimal for mounting on a mobile robot that projects an interactive window as shown in FIG. 2 on a floor surface, a wall surface, a ceiling surface, or the like. Here, the user performs a mouse click-like operation on the projected screen using a light emitting pointer such as a laser pointer or a flashlight. The light source of the light emitting pointer is preferably a laser, but is not particularly limited.

  Specifically, for example, the selection button “instruction 1” in FIG. 2 is selected by the mouse click-like operation, and then the selection button “OK” is selected by the mouse click-like operation. As a result, the selection button “instruction 1” is formally selected and executed. Here, in order to prevent erroneous input, it is preferable to use the two-stage method in this way. However, the operation of selecting the selection button “OK” may be omitted, and the selection button “instruction 1” may be formally selected by selecting only the selection button “instruction 1”.

The controller 101 instructs the dialog window generation unit 102 to create a dialog window. Then, the controller 101 instructs the projector 103 to project the dialogue window created by the dialogue window generation unit 102.
On the other hand, the controller 101 defines an operation in accordance with an instruction (for example, the selection buttons “instruction 1” and “instruction 2” illustrated in FIG. 2) input via the bright spot detection unit 108 and the fluctuation detection unit 110. Instruct other units (not shown).

The dialogue window generation unit 102 creates a dialogue window based on an instruction from the controller 101. The created dialogue window may be projected directly by the projector 103 or may be projected once stored in a buffer (not shown).
The projector 103 projects an interactive window on a predetermined location based on an instruction from the controller 101.

The camera 104 captures the background before projecting the dialogue window and the projected dialogue window. Here, while the mobile robot is activated, the camera 104 shoots a video at any time. Of course, for example, a configuration may be adopted in which shooting is performed only when necessary based on an instruction from the controller 101.
An image captured by the camera 104 is stored in the image buffer 105.

  The frame comparison unit 106 compares the dialog window image stored in the image buffer 105 with the background image, and performs background image correction. FIG. 3 is a conceptual diagram of background image correction. As shown in FIG. 3, the background image correction is a correction for removing the background image data from the image data of the dialogue window including the background image. In a mobile robot, the robot itself projects a dialogue window while moving. Therefore, it is not necessarily projected onto a white screen. Specifically, projection is performed on various objects in a real environment such as a floor surface, a wall surface, and a ceiling surface. In the present invention, by performing background image correction, it is possible to accurately detect the movement of a bright spot at any projection location.

  Further, the frame comparison unit 106 corrects the trapezoidal shape and rotation of the captured dialogue window based on an instruction from the region calibration unit 107. Specifically, as shown in FIG. 3, for example, the dialogue window projected on the floor is distorted in a trapezoidal shape.

  Further, when projecting a dialog window to a user facing the robot, the dialog window is rotated 180 degrees and projected. Therefore, it is necessary to correct the trapezoidal shape and rotation of the dialogue window. FIG. 3 shows a case where the dialog window shown in FIG. 2 is projected after being rotated by 180 degrees.

  The bright spot detection unit 108 detects the bright spot indicated by the user with the pointer while referring to the bright spot stationary timer 109. The bright spot detection unit 108 calculates the center of gravity of the detected bright spot. Details of the bright spot detection method will be described later.

  The bright spot fluctuation detection unit 110 includes a low pass filter (LPF) that detects fluctuations of the bright spot pointed to by the user with the pointer. Here, the position of the bright spot pointed to by the user fluctuates. On the other hand, background reflections and external light are stationary. Therefore, light other than the bright spot indicated by the user can be removed as a false bright spot.

  Further, the bright spot fluctuation detection unit 110 determines that the bright spot is stationary when the fluctuation of the bright spot center of gravity is within the set region. Then, if the luminescent spot is stationary for a predetermined time, the selection button of the dialogue window where the luminescent spot is located is selected. This instruction is input to the controller 101. On the other hand, if it is determined that the bright spot is not stationary, the stationary timer 109 is reset.

  As described above, according to the configuration of the present invention, it is possible to provide a projection type user interface system capable of mouse click-like operation only by light irradiation with a pointer to a projected dialogue window, that is, simply and at low cost. .

Next, a processing flow of the projection type user interface system will be described with reference to FIG.
First, the dialogue window generation unit 102 creates a dialogue window based on an instruction from the controller 101 (S201).
Next, based on the instruction from the controller 101, the dialogue window created by the projector 103 is projected onto a floor surface, a wall surface, or the like (S202). Note that the background is photographed by the camera 104 before the dialogue window is projected.

Next, the projected dialogue window is photographed by the camera 104 (S203).
Next, the frame comparison unit 106 performs background image correction for removing the background photographed before projection from the photographed dialogue window image (S204).
Subsequently, the frame comparison unit 106 corrects the trapezoidal shape and rotation of the captured dialogue window based on an instruction from the region calibration unit 107 (S205).

  Next, the bright spot detection unit 108 searches for the bright spot of the pointer pointed by the user (S206). Specifically, based on the image after the background image correction, the binarization cut-off level is swept, and it is determined whether or not a bright area of a certain size (for example, 32 × 32 pixels) remains.

If there is a bright spot (YES in S207), the center of gravity of the bright spot is calculated, and the movement of the center of bright spot is detected (S208).
On the other hand, if there is no bright spot (NO in S207), the process returns to S203, and the projected conversation window is captured by the camera 104.

  Next, the bright spot fluctuation detection unit 110 determines whether or not the position of the center of gravity fluctuates (S209). When the bright spot does not fluctuate (S209 NO), it is determined that it is a false bright spot, and the process returns to S206 to search for the bright spot again. On the other hand, when the bright spot fluctuates (YES in S209), it is determined that it is a bright spot. Then, it is determined whether the locus of the center of gravity is equal to or less than a threshold value (S210).

  When the locus of the center of gravity is not less than or equal to the threshold (NO in S210), the bright spot fluctuation detection unit 110 determines that the bright spot is not stationary. Then, the stationary timer 109 is reset, the process returns to S208, and the movement of the bright spot centroid is detected again. On the other hand, when the locus of the center of gravity is equal to or smaller than the threshold (S210 YES), it is determined that the bright spot is stationary. Then, it is determined whether or not the stationary time is being measured (S211).

  When the stationary time is not being measured (NO in S211), the fluctuation detecting unit 110 starts the stationary timer 109, returns to S208, and detects the movement of the bright spot centroid. On the other hand, when the stationary time is being measured (YES in S211), it is subsequently determined whether or not the stationary time exceeds the set value (S212).

  If the stationary time does not exceed the set value (NO in S212), the process returns to S208, and the movement of the bright spot centroid is detected. On the other hand, if the stationary time exceeds the set value (S212 YES), trigger coordinates are output (S213). The selection button selected from this coordinate is recognized. As described above, a mouse click-like operation can be performed simply by irradiating the projected dialogue window with light using a pointer without using a complicated mechanism such as wireless communication. That is, a mouse click-like operation can be performed easily and at low cost.

  The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention already described.

It is a block diagram of the projection type user interface system concerning an embodiment of the invention. It is a figure which shows an example of a dialogue window. It is a conceptual diagram of background image correction. It is a flowchart of the projection type user interface system concerning an embodiment of the invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Controller 102 Dialog window generation part 103 Projector 104 Camera 105 Image buffer 106 Image comparison part 107 Area correction part 108 Bright spot detection part 109 Bright spot stationary timer 110 Bright spot fluctuation detection part

Claims (7)

A projector that projects a dialogue window;
A camera that captures the dialogue window projected by the projector;
A bright spot detection unit that detects a movement of the center of gravity of a bright spot by a pointer on the dialog window from an image of the dialog window captured by the camera;
A projection type user interface system comprising: a bright spot fluctuation detecting unit that detects fluctuation of the center of gravity of the bright spot.   The projection type user interface system according to claim 1, wherein the bright spot fluctuation detecting unit includes a low-pass filter.   3. The projection type user interface system according to claim 1, further comprising an image comparison unit that removes a background image of a projection location of the dialogue window captured by the camera from the image of the dialogue window.   The projection type user interface system according to any one of claims 1 to 3, wherein when the fluctuation of the bright spot is within a predetermined region, the bright spot is determined to be stationary.   The projection type user interface system according to any one of claims 1 to 4, wherein when the bright spot is stationary for a predetermined time, it is determined that a selection button on the dialogue window where the bright spot is located is selected. .   The projection type user interface system according to any one of claims 1 to 5, wherein when the fluctuation of the bright spot is not detected, the bright spot is determined not to be a bright spot by a pointer.   A mobile robot equipped with the projection type user interface system according to claim 1.

Images