JP2018147307A - Display operation system, control method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display operation system capable of preventing the occurrence of an error operation to an operation component of an application which is projected and displayed.SOLUTION: In a camera scanner 101, when a short focus projector 207 is turned on again after it is turned off, a user gesture operation for a GUI component projected to and displayed on an operation plane is ignored in the case where an object is not removed from the operation plane of a document table 204.SELECTED DRAWING: Figure 10

Description

  The present invention relates to a display operation system that recognizes an operation on an operation component of a displayed application by detecting a user's gesture operation, a control method thereof, and a program.

  There is known a display operation system that recognizes an operation on an operation component including a graphical user interface of a projected application by detecting a movement (gesture operation) of a user's hand or a fingertip (for example, see Patent Document 1). .) A display operation system that recognizes an operation on an operation component of an application projected and displayed by detecting a three-dimensional coordinate of an object on a document table, for example, a pen, is not limited to the movement of a user's hand or fingertip. ing.

  In such a display operation system, it is possible to perform an explanation using a document, a signature on the document, an explanation using a projected display application, and an operation on an operation component of the projected display using the same document stand. . Thereby, for example, the business efficiency of a salesperson as a user can be improved, and the work efficiency of confirmation and signature of a purchaser or contractor as a user can be improved. In addition, such a display operation system has a blank function for temporarily interrupting projection by the projector, and includes a projector turn-off key corresponding to the blank function. For example, when giving a written explanation or signing a document, in order to avoid the operation parts of the application being superimposed on the document, the user operates the extinction key to display the projection of the operation parts of the application by the projector. Interrupt.

Japanese Patent Laying-Open No. 2005-267034

  However, after completing the explanation using the document and signing the document, the user can unintentionally operate the application again by operating the extinction key while keeping his hand at the position corresponding to the operation part of the application. Parts may be projected and displayed. In addition, the operation component of the application may be projected and displayed again by operating the extinction key while the pen used for the signature is left in a position corresponding to the operation component of the application. In such a case, there is a risk that it is recognized that an operation on the operation component of the application has been performed by the user's hand or pen. That is, there is a possibility that an erroneous operation on the operation component of the projected application is generated.

  An object of the present invention is to provide a display operation system, a control method thereof, and a program capable of preventing an erroneous operation on an operation component of an application displayed by projection.

  In order to achieve the above object, a display operation system of the present invention includes a display unit that projects and displays an operation component of an application on an operation surface, a detection unit that detects an object existing on the operation surface, and the operation component. And a control unit that controls the operation of the application based on an operation on the display unit when the display unit displays the operation component on the operation surface on which the operation component is not displayed. When the detection unit detects the object, the control unit ignores an operation on the operation component.

  According to the present invention, it is possible to prevent an erroneous operation on an operation component of an application that is projected and displayed.

1 is a diagram showing a network configuration including a camera scanner as a display operation system according to a first embodiment of the present invention. It is a figure which shows schematically the structure of the camera scanner in FIG. It is a figure for demonstrating the coordinate system in the camera scanner of FIG. The relationship between the orthogonal coordinate system, the camera coordinate system, and the camera imaging plane in FIG. 3 is shown. It is a block diagram which shows the hardware constitutions of the controller part of the camera scanner of FIG. FIG. 6 is a block diagram showing the configuration of functional modules of a camera scanner control program executed by the CPU in FIG. 5. It is a figure which shows the hardware constitutions of the distance image sensor part in FIG. FIG. 3 is a diagram for explaining a problem that may occur when a GUI part is projected and displayed on an operation plane of a document table on which no GUI part is projected and displayed in the camera scanner of FIG. 2. It is a flowchart which shows the projector light extinction / relighting process as a control method of the display operation system which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the projector light extinction / relighting process as a control method of the display operation system which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the power-on process as a control method of the display operation system which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the projector light extinction / relighting process as a control method of the display operation system which concerns on the 3rd Embodiment of this invention. It is a figure which shows an example of the object removal screen displayed in step S1201 of FIG. It is a flowchart which shows the power-on process as a control method of the display operation system which concerns on the 4th Embodiment of this invention. It is a flowchart which shows the screen display process as a control method of the display operation system which concerns on the 5th Embodiment of this invention. It is a figure which shows an example of the screen for demonstrating operation prohibition mode. It is a flowchart which shows the GUI component operation permission process as a control method of the display operation system which concerns on the 5th Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments. First, a first embodiment of the present invention will be described.

  FIG. 1 is a diagram showing a network configuration including a camera scanner as a display operation system according to a first embodiment of the present invention. In FIG. 1, a camera scanner 101 is connected to a host computer 102 and a printer 103 via a network 104 such as Ethernet (registered trademark). In this embodiment, in response to an instruction from the host computer 102, a scan function for reading an image by the camera scanner 101 and a print function for outputting the scan data generated by the scan function by the printer 103 are executed. Further, by directly instructing the camera scanner 101 without using the host computer 102, the scan function and the print function can be executed.

  FIG. 2 is a diagram schematically showing the configuration of the camera scanner in FIG. 2, the camera scanner 101 includes a controller unit 201, a camera unit 202, an arm unit 203, a short focus projector 207, and a distance image sensor unit 208 as hardware devices. A controller unit 201 constituting the main body of the camera scanner 101, a camera unit 202 for imaging, a short focus projector 207, and a distance image sensor unit 208 including a three-dimensional distance sensor are connected to each other by an arm unit 203. The arm portion 203 has a plurality of joints, and is configured to be stretchable by being bent at each joint. The controller unit 201 includes a power key 210 that controls on / off of the power of the camera scanner 101 and a light-off key 211 that controls start and stop of projection by the short focus projector 207.

  The camera scanner 101 is disposed beside a document table 204 having an operation plane (operation surface). The camera unit 202 and the distance image sensor unit 208 are directed toward the document table 204, and the camera unit 202 reads an image in the reading area 205 on the operation plane surrounded by a broken line in the drawing. For example, the camera unit 202 reads an image of the document 206 placed in the reading area 205. A turntable 209 is provided in the document table 204. The turntable 209 is rotated by an instruction from the controller unit 201, and the relative angle between the object placed on the turntable 209 and the camera unit 202 can be changed. In the camera scanner 101, the camera unit 202 and the distance image sensor unit 208 constitute a detection unit that detects an object existing on the operation plane of the document table 204. In the camera scanner 101, the camera unit 202 includes a camera that captures an image at a single resolution. However, the camera unit 202 may be a camera that can switch between high-resolution image capturing and low-resolution image capturing. The camera scanner 101 may include an LCD touch panel (not shown) arranged on the document table 204 and a speaker 514 described later. Further, the camera scanner 101 may include various sensor devices such as a human sensor, an illuminance sensor, and an acceleration sensor for collecting surrounding environment information in addition to the distance image sensor unit 208.

  FIG. 3 is a diagram for explaining a coordinate system in the camera scanner of FIG. In FIG. 3, the camera scanner 101 defines a camera coordinate system, a distance image coordinate system, and a projector coordinate system. The camera coordinate system is a coordinate system in which an image plane captured by the camera unit 202 is defined as an XY plane and a direction orthogonal to the image plane is defined as a Z direction. The distance image coordinate system is a coordinate system in which an image plane captured by a later-described RGB camera 701 included in the distance image sensor unit 208 is defined as an XY plane, and a direction orthogonal to the image plane is defined as a Z direction. The projector coordinate system is a coordinate system in which an image plane on which the short focus projector 207 projects an image is defined as an XY plane, and a direction orthogonal to the image plane is defined as a Z direction. Further, in the camera scanner 101, in order to be able to handle three-dimensional data of these three independent coordinate systems in a unified manner, the plane including the document table 204 is defined as an XY plane, and a direction orthogonal to the XY plane is set. An orthogonal coordinate system defined as the Z direction is defined.

FIG. 4 shows the relationship between the orthogonal coordinate system, the camera coordinate system, and the camera imaging plane in FIG. In the camera scanner 101, the three-dimensional point P [X, Y, Z] in the orthogonal coordinate system can be converted into the three-dimensional point Pc [Xc, Yc, Zc] in the camera coordinate system by the following equation (1).
[Xc, Yc, Zc] T = [Rc | tc] [X, Y, Z, 1] T (1)
Here, Rc is a 3 × 3 rotation matrix, and tc is a translation vector. Rc and tc are configured by external parameters determined based on the posture (rotation) and position (translation) of the camera with respect to the orthogonal coordinate system. Further, the three-dimensional point Pc [Xc, Yc, Zc] defined in the camera coordinate system can be converted into the three-dimensional point P [X, Y, Z] in the orthogonal coordinate system by the following equation (2).
[X, Y, Z] T = [Rc-1 | -Rc-1tc] [Xc, Yc, Zc, 1] T (2)

A two-dimensional image plane (hereinafter referred to as “camera imaging plane”) photographed by the camera unit 202 is converted by the camera unit 202 from three-dimensional information of a three-dimensional point group in the three-dimensional space into two-dimensional information. Consists of. That is, the camera imaging plane is configured by perspective-projecting the three-dimensional point Pc [Xc, Yc, Zc] on the camera coordinate system into the two-dimensional coordinate pc [xp, yp] by the following equation (3). Can do.
λ [xp, yp, 1] T = A [Xc, Yc, Zc, 1] T (3)
Here, A is a camera internal parameter which is a 3 × 3 matrix expressed by the focal length and the image center.

  As described above, the camera scanner 101 converts the three-dimensional point group represented by the orthogonal coordinate system into the three-dimensional point group or the camera imaging plane in the camera coordinate system by using the above formulas (1) and (3). can do. In the camera scanner 101, the internal parameters of each hardware device and the position and orientation (external parameters) with respect to the orthogonal coordinate system are calibrated in advance by a known calibration method. Hereinafter, in this specification, the “three-dimensional point group” indicates a three-dimensional point group represented by an orthogonal coordinate system.

  FIG. 5 is a block diagram showing a hardware configuration of the controller unit of the camera scanner of FIG. In FIG. 5, the controller unit 201 includes a CPU 502, a RAM 503, a ROM 504, an HDD 505, a network I / F 506, and an image processing processor 507 connected to a system bus 501. The controller unit 201 further includes a camera I / F 508, a display controller 509, a serial I / F 510, an audio controller 511, and a USB controller 512. The CPU 502 is a central processing unit that controls the operation of the entire controller unit 201. The RAM 503 is a volatile memory. A ROM 504 is a nonvolatile memory, and stores a startup program for the CPU 502. The HDD 505 is a hard disk drive (HDD) having a larger capacity than the RAM 503. The HDD 505 stores a control program for the camera scanner 101 executed by the controller unit 201.

  The CPU 502 executes an activation program stored in the ROM 504 when the camera scanner 101 is activated such as when the power is turned on. This activation program reads the control program stored in the HDD 505 and expands it in the RAM 503. When the CPU 502 executes the startup program, it continuously executes the control program developed in the RAM 503 to control the operation of the entire controller unit 201. The CPU 502 also stores data used for execution of the control program in the RAM 503 to read / write. The HDD 505 can store various settings necessary for execution of the control program and image data generated by the camera unit 202 through imaging. The stored data is read and written by the CPU 502. Further, the CPU 502 communicates with other devices connected to the network 104 via the network I / F 506.

  The serial I / F 510 inputs / outputs serial signals. In the controller unit 201, when the serial I / F 510 is connected to the hard key I / F 513 and the power key 210 or the turn-off key 211 is pressed, the serial I / F 510 notifies the CPU 502 of a signal corresponding to the pressed key. . When the power of the controller unit 201 is off, the hard key I / F 513 turns on a controller power (not shown) when the power key 210 is pressed. As a result, the above-described activation program is executed by the CPU 502 described above. The image processor 507 reads out the image data stored in the RAM 503, performs image processing, and writes it back into the RAM 503 again. Note that image processing executed by the image processor 507 includes rotation, scaling, color conversion, and the like. The camera I / F 508 is connected to the camera unit 202 and the distance image sensor unit 208, acquires image data from the camera unit 202 in response to an instruction from the CPU 502, acquires distance image data from the distance image sensor unit 208, and stores the image data in the RAM 503. Write. In addition, the camera I / F 508 transmits a control command from the CPU 502 to the camera unit 202 and the distance image sensor unit 208 to set the camera unit 202 and the distance image sensor unit 208. It should be noted that at least one of the display controller 509, the serial I / F 510, the audio controller 511, and the USB controller 512 may be included in the controller unit 201.

  A display controller 509 controls display of image data on a display (not shown) in accordance with an instruction from the CPU 502. The display controller 509 is connected to the short focus projector 207. The audio controller 511 is connected to the speaker 514, converts audio data into an analog audio signal in accordance with an instruction from the CPU 502, and outputs audio through the speaker 514. The USB controller 512 controls an external USB device in accordance with an instruction from the CPU 502. The USB controller 512 is connected to an external memory 515 such as a USB memory or an SD card, and reads / writes data from / to the external memory 515.

  FIG. 6 is a block diagram showing the configuration of the functional modules of the camera scanner control program executed by the CPU in FIG. The control program for the camera scanner 101 is stored in the HDD 505 as described above, and the CPU 502 develops the control program in the RAM 503 and executes it when activated. A functional configuration 601 is configured when the control program is executed. The functional configuration 601 includes a main control unit 602, an image acquisition unit 603, a camera image acquisition unit 604, a distance image acquisition unit 605, a recognition processing unit 606, an operation recognition unit 607, a gesture recognition unit 608, an object detection unit 609, and the like. An image processing unit 610 is provided. Furthermore, the functional configuration 601 includes a key processing unit 611, a power control unit 612, a display control unit 613, an application unit 614, a GUI control unit 615, a display unit 616, and a network communication unit 617 as modules.

  The main control unit 602 is a central module of control, and controls other modules included in the functional configuration 601. The image acquisition unit 603 is a module that performs image input processing, and includes a camera image acquisition unit 604 and a distance image acquisition unit 605. The camera image acquisition unit 604 acquires image data output from the camera unit 202 via the camera I / F 508 and stores the image data in the RAM 503. The distance image acquisition unit 605 acquires the distance image data output from the distance image sensor unit 208 via the camera I / F 508 and stores it in the RAM 503. The recognition processing unit 606 is a module that detects and recognizes the movement of an object on the operation plane of the document table 204 from the image data acquired by the camera image acquisition unit 604 and the distance image data acquired by the distance image acquisition unit 605. The recognition processing unit 606 includes an operation recognition unit 607, a gesture recognition unit 608, and an object detection unit 609. The gesture recognition unit 608 continues to acquire an image on the operation plane of the document table 204 from the image acquisition unit 603, and detects a user's gesture operation such as touch. Specifically, the gesture recognition unit 608 acquires a three-dimensional point cloud of an object existing on the operation plane of the document table 204 from the distance image data acquired by the distance image acquisition unit 605, and detects the outer shape of the object. Furthermore, the gesture recognition unit 608 determines the position of the user's hand and fingertip from the curvature of each part of the outer shape, and detects a gesture operation using one user's finger, for example. The gesture recognizing unit 608 can detect various other gesture operations. For example, the gesture recognition unit 608 can detect a gesture operation using the user's fingers or hands, or the user's arm or the entire body. it can. The operation recognition unit 607 notifies the main control unit 602 when the gesture recognition unit 608 detects a user's gesture operation. The object detection unit 609 acquires an image obtained by imaging the operation plane of the document table 204 from the image acquisition unit 603 and detects an object existing on the operation plane of the document table 204. The object detection unit 609 also detects the timing at which an object is placed on the operation plane of the document table 204, the timing at which the object is placed and stopped, or the timing at which the object is removed.

  The image processing unit 610 is used by the image processing processor 507 to analyze the images acquired from the camera unit 202 and the distance image sensor unit 208, and includes various image processing modules. The gesture recognition unit 608 and the object detection unit 609 described above are realized using various image processing modules of the image processing unit 610.

  The key processing unit 611 is a module that executes processing in response to pressing of the power key 210 and the turn-off key 211 notified via the hard key I / F 513, and includes a power control unit 612 and a display control unit 613. When the power control unit 612 is notified of pressing of the power key 210 when the controller power (not shown) is turned on, the power control unit 612 executes the end processing of the program executed by the CPU 502, turns off the controller power, and performs the power off processing. Do. Further, when the controller 612 is notified that the power key 210 is pressed when the controller power is off, the power controller 612 directly turns on the controller power from the hard key I / F 513. When notified that the extinguishing key 211 is pressed, the display control unit 613 determines whether or not the short focus projector 207 is turned on via the serial I / F 510. If the short focus projector 207 is turned on, the display control unit 613 turns off the short focus projector 207 via the serial I / F 510 and interrupts the projection display. In addition, if the short focus projector 207 is not turned on, the display control unit 613 turns on the short focus projector 207 via the serial I / F 510 and executes projection display. Further, the display control unit 613 notifies the main control unit 602 whether the short focus projector 207 is turned on or off, and the main control unit 602 displays the application unit 614 according to whether the short focus projector 207 is turned on or off. Control the display of the current screen.

  The application unit 614 holds data of a plurality of screens constituting an application operated by a user and data of a graphical user interface (GUI) component constituting each screen, and further performs screen transition for transitioning between the plurality of screens. Run. Further, the application unit 614 defines a process to be executed when a GUI component is operated, and stores it in association with data of the GUI component. The application unit 614 transmits the GUI component data constituting the screen to the GUI control unit 615 and requests the GUI control unit 615 to display the screen. The GUI control unit 615 generates corresponding GUI components (operation components) such as messages and buttons in response to requests from the main control unit 602 and the application unit 614. In addition, the process executed when the generated GUI component is operated is associated with the GUI component. Here, the process associated with the GUI component is, for example, a display change process for changing the color or shape of the GUI part, a screen transition process for transitioning each screen constituting the application, or a programmed process or an external program. This corresponds to execution processing to be started. Furthermore, the GUI control unit 615 requests the display unit 616 to display the generated GUI component. The display unit 616 causes the short focus projector 207 to display the requested GUI component via the display controller 509. Since the short focus projector 207 is installed toward the operation plane of the document table 204, a GUI component is projected and displayed on the operation plane of the document table 204. In addition, the GUI control unit 615 receives a gesture operation such as a user's touch recognized by the gesture recognition unit 608 and coordinates where the gesture operation is performed via the main control unit 602. In addition, the GUI control unit 615 determines the GUI component operated by associating the coordinates of the projected GUI component with the operation coordinates, and the operation content, and executes a process associated with the GUI component. The network communication unit 617 performs communication by TCP / IP with other devices connected to the network 104 via the network I / F 506.

  FIG. 7 is a diagram illustrating a hardware configuration of the distance image sensor unit in FIG. In FIG. 7, the distance image sensor unit 208 includes an infrared pattern projection type three-dimensional distance image sensor, and includes an RGB camera 701, an infrared pattern projection unit 702, and an infrared camera 703. The RGB camera 701 captures visible light with RGB signals, and the infrared pattern projection unit 702 projects a three-dimensional measurement pattern 706 onto the object 704 with infrared light that is invisible light. The infrared camera 703 reads the three-dimensional measurement pattern 706 projected onto the object. The distance image sensor unit 208 associates the three-dimensional measurement pattern 706 with the captured image 707 of the infrared camera 703. Further, the distance image sensor unit 208 calculates the distance of each pixel of the captured image 707 from the infrared camera 703 by using the principle of triangulation with the straight line 705 connecting the infrared pattern projection unit 702 and the infrared camera 703 as a base line. . Thereby, the distance image sensor unit 208 generates distance image data in which each pixel has a distance value. In the present embodiment, an infrared pattern projection type three-dimensional distance image sensor is used as the distance image sensor unit 208, but another type of distance image sensor may be used. For example, a distance image sensor using a stereo system that performs stereo stereoscopic vision with two RGB cameras, a TOF (Time of Flight) system that measures distance by detecting the flight time of laser light, or the like may be used.

  FIG. 8 is a diagram for explaining a problem that may occur when a GUI part is projected and displayed on the operation plane of the document table on which the GUI part is not projected and displayed in the camera scanner of FIG. For example, as shown in FIG. 8A, in the camera scanner 101, the display screen projected by the display unit 616 on the operation plane of the document table 204 has a button 802 and a button 803 as GUI components in addition to the message 801. The case will be described. At this time, when the operation recognizing unit 607 recognizes the touch of the button 802 or the button 803 by the user through the detection of the object by the camera unit 202 and the distance image sensor unit 208, the GUI control unit 615 is associated with the button 802 or the button 803. Execute the process. In the example of FIG. 8A, the button 802 and the button 803 are associated with the transition of the display screen.

  When the user presses the turn-off key 211, the display control unit 613 turns off the short focus projector 207, and the projection display on the display screen is interrupted as shown in FIG. 8B. Since the execution of the application is interrupted when the projection display is interrupted, the execution of the process associated with the button 802 or the button 803 must be prohibited. Therefore, after the projection display on the display screen is interrupted, the detection of an object existing on the operation plane of the document table 204 is stopped so that the user's touch of the button 802 or the button 803, that is, the user's gesture operation cannot be recognized. To do. That is, the camera unit 202 and the distance image sensor unit 208 are turned off.

  Next, after the user performs an operation such as signing a document placed on the operation plane of the document table 204, the pen 804 or the hand 805 is placed on the operation plane (FIG. 8C) and left as it is. The user presses the turn-off key 211. At this time, the display control unit 613 turns on the short focus projector 207 and projects and displays a display screen similar to that shown in FIG. Here, since execution of the application is resumed, execution of processing associated with the button 802 and the button 803 must also be permitted. Therefore, after the display screen is projected and displayed, the detection of an object existing on the operation plane of the document table 204 is resumed so that the touch of the button 802 or the button 803 by the user can be recognized. That is, the camera unit 202 and the distance image sensor unit 208 are turned on.

  However, as shown in FIG. 8D, when the pen 804 or the hand 805 is left on the button 802 or the button 803, the operation recognition unit 607 recognizes that the button 802 or the button 803 is touched by the user. . The user may leave the pen 804 or the hand 805 on the operation plane without intending to change the display screen. In this case, the user is associated with the button 802 or the button 803 regardless of the user's intention. Transition of the display screen is executed. That is, an operation (erroneous operation) of the button 802 or the button 803 that is not intended by the user occurs. In the present embodiment, corresponding to this, even after the display screen is projected and displayed, the user's touch on the button 802 or the button 803 by the operation recognition unit 607 is ignored under certain conditions.

  9 and 10 are flowcharts showing projector turn-off / re-lighting processing as a control method of the display operation system according to the present embodiment. 9 and 10, the user presses the turn-off key 211 to turn off the short focus projector 207, and then the user presses the turn-off key 211 again to display the display screen from the short focus projector 207 to the document stand 204. The main control unit 602 executes the projection display.

  First, when the user presses the turn-off key 211 while the display screen is projected and displayed from the short focus projector 207 to the document stage 204, the display control unit 613 notifies the main control unit 602 that the turn-off key 211 has been pressed. In response to the notification of pressing of the turn-off key 211, the main control unit 602 first turns off the short focus projector 207 via the display control unit 613 (step S901). Further, the main control unit 602 performs object detection processing on the operation plane of the document table 204 by the object detection unit 609, and detects an object existing on the operation plane (step S902). Next, the main control unit 602 interrupts execution of the application and turns off the distance image sensor unit 208 so that the touch by the user cannot be recognized (step S903). Accordingly, the user's gesture operation cannot be accepted while the application is not being executed. In the subsequent step S904, the main control unit 602 determines again whether or not the turn-off key 211 is pressed by the user, and repeats the determination in step S904 while the turn-off key 211 is not pressed. When the turn-off key 211 is pressed, the main control unit 602 resumes execution of the application and then turns on the distance image sensor unit 208 again (step S905). After that, the main control unit 602 performs object detection processing on the operation plane of the document table 204 by the object detection unit 609 as in step S902, and detects an object present on the operation plane (step S906). Next, the main control unit 602 relights the short focus projector 207 via the display control unit 613 (step S901). A series of processing from step S901 to step S907 in FIG. 9 is hereinafter defined as “main processing S908”.

  In step S909, by comparing the object detected in step S902 with the object detected in step S906, the object existing on the operation plane is changed between the time when the short focus projector 207 is turned off and the light is turned on again. Determine whether or not. The “change in the object” in the present embodiment means a change (movement) of the position of the object, a change in the shape of the object, and removal or addition of the object. As a result of the determination in step S909, when the object has not changed, the main control unit 602 displays the GUI component on the GUI control unit 615 using the GUI component data transmitted from the application unit 614 to the GUI control unit 615. This is executed (step S910). After that, when the operation recognizing unit 607 recognizes the user's gesture operation through the distance image sensor unit 208 turned on again in step S905, the main control unit 602 permits notification of the recognized gesture operation to the GUI control unit 615. (Step S911). As a result, the GUI component can accept the user's gesture operation. That is, the user's gesture operation is not ignored. Thereafter, this process is terminated.

  As a result of the determination in step S909, even when the operation recognition unit 607 recognizes the user's gesture operation when the object has changed, the main control unit 602 notifies the GUI control unit 615 of the recognized gesture operation. It is prohibited (step S912). As a result, the GUI component cannot accept the user's gesture operation, and the user's gesture operation is ignored. Next, the main control unit 602 causes the GUI control unit 615 to display a GUI component, similarly to step S910 (step S913). At this time, although the GUI component is projected and displayed on the operation plane of the document table 204, the user's gesture operation on the GUI component is ignored. After that, the main control unit 602 executes object detection processing on the operation plane of the document table 204 by the object detection unit 609, similarly to step S902 and step S906, and detects an object existing on the operation plane (step S914). ). Next, in step S915, it is determined whether or not an object is detected in step S914. Further, in step S915, by comparing the object detected in step S902 with the object detected in step S914, the object existing on the operation plane changes between the time when the short focus projector 207 is turned off and the light is turned on again. Determine whether or not. By the way, when executing step S915, it is assumed that the object existing on the operation plane has been changed in step S909. Therefore, when the object is turned off when the short focus projector 207 is turned off at the time of executing step S915. It is unlikely to return to this state. Accordingly, in step S915, it is determined whether or not an object existing on the operation plane has been substantially detected after the short focus projector 207 is turned on again, that is, whether or not the object has been removed from the operation plane. ing.

  As a result of the determination in step S915, when an object is detected (that is, when the object is not removed from the operation plane), step S914 is repeated. When the object is not detected (that is, when the object is removed from the operation plane), the process proceeds to step S911. Accordingly, the GUI component can accept the user's gesture operation, and the user's gesture operation is not ignored. Thereafter, this process is terminated.

  9 and 10, when the short focus projector 207 is turned on again after the short focus projector 207 is turned off and the object is not removed from the operation plane, the user's gesture operation on the GUI component is ignored. Is done. Thus, for example, even if the GUI part of the application is projected and displayed again in a state where the user does not intend to operate the GUI part and the hand or pen is left at a position corresponding to the GUI part, the hand or pen Operations on GUI parts by are ignored. Thereby, it is possible to prevent an erroneous operation on the GUI component by a hand or a pen placed unintentionally for the operation of the GUI component.

  9 and 10, the object detection unit 609 operates on the operation plane of the document table 204 after the short focus projector 207 is turned off until the short focus projector 207 is turned on again. Execute object detection processing. More specifically, the object detection unit 609 performs object detection processing before the distance image sensor unit 208 is turned off and after the distance image sensor unit 208 is turned on again, and compares the detection results. (Step S909). Accordingly, after the short focus projector 207 is turned off and the distance image sensor unit 208 is turned off, the user can reliably detect the hand or pen placed on the operation plane of the document table 204 without intending to operate the GUI component. can do.

  By the way, it is conceivable that before the short focus projector 207 is turned off, the user places not only a hand and a pen but also an object used for the operation on the operation plane of the document table 204 in order to operate the GUI component. In this case, the user is less likely to move (change) the placed object. Correspondingly, in the processes of FIGS. 9 and 10, if the object present on the operation plane does not change between the time when the short focus projector 207 is turned off and the time when it is turned on again (NO in step S909), the gesture for the GUI component is performed. The operation is not ignored. Thereby, the process linked | related with GUI components can be performed as a user's intent, and the convenience with respect to a user can be maintained.

  It is also conceivable that a plurality of GUI parts are projected and displayed on the operation plane of the document table 204 when the short focus projector 207 is turned on. In such a case, in the processes of FIGS. 9 and 10, if the object is not removed from the operation plane even after the short focus projector 207 is turned on again, the gesture operation for all the GUI parts is ignored. The same applies to the second to fourth embodiments described below.

  Next, a second embodiment of the present invention will be described. Since the configuration and operation of the second embodiment are basically the same as those of the first embodiment described above, the description of the overlapping configuration and operation will be omitted, and different configurations and operations will be described below. Give an explanation.

  When a user places a hand or a pen at a position corresponding to a GUI part without intending to operate the GUI part, the short focus projector 207 is turned off and a document placed on the operation plane of the document table 204 is signed. In addition to the case, the camera scanner 101 may be activated. In the present embodiment, in response to this, when the camera scanner 101 is activated, the user's gesture operation on the GUI component is ignored under certain conditions.

  FIG. 11 is a flowchart showing a power-on process as a control method of the display operation system according to the present embodiment. The process in FIG. 11 is executed by the main control unit 602 when the user presses the power key 210 in the controller unit 201 of the camera scanner 101.

  First, when the power of the controller unit 201 is off and the power key 210 is pressed, the main control unit 602 turns on the distance image sensor unit 208 (step S1101). Thereafter, the main control unit 602 performs object detection processing on the operation plane of the document table 204 by the object detection unit 609, and detects an object existing on the operation plane (step S1102). Next, the main control unit 602 turns on the short focus projector 207 via the display control unit 613 (step S1103). Further, the main control unit 602 activates the application unit 614 (step S1104). Note that a series of processing from step S1101 to step S1104 in FIG. 11 is hereinafter defined as “startup processing S1105”.

  In step S1106, the main control unit 602 determines whether an object present on the operation plane of the document table 204 is detected in step S1102. If the object is not detected, the main control unit 602 causes the GUI control unit 615 to display the GUI component using the GUI component data transmitted from the application unit 614 to the GUI control unit 615 (step S1107). After that, when the operation recognition unit 607 recognizes the user's gesture operation through the distance image sensor unit 208 turned on in step S1101, the main control unit 602 permits notification of the recognized gesture operation to the GUI control unit 615. (Step S1108). As a result, the GUI component can accept the user's gesture operation. That is, the user's gesture operation is not ignored. Thereafter, this process is terminated.

  If the object is detected as a result of the determination in step S1106, the main control unit 602 notifies the GUI control unit 615 of the recognized gesture operation even if the operation recognition unit 607 recognizes the gesture operation of the user. It is prohibited (step S1109). As a result, the GUI component cannot accept the user's gesture operation, and the user's gesture operation is ignored. Next, the main control unit 602 causes the GUI control unit 615 to display a GUI component as in step S1107 (step S1110). At this time, although the GUI component is projected and displayed on the operation plane of the document table 204, the user's gesture operation on the GUI component is ignored. After that, the main control unit 602 executes object detection processing on the operation plane of the document table 204 by the object detection unit 609 as in step S1102, and detects an object existing on the operation plane (step S1111). In step S1112, the main control unit 602 determines whether an object present on the operation plane of the document table 204 is detected in step S1111.

  If the object is detected as a result of the determination in step S1112 (that is, if the object is not removed from the operation plane), step S1110 is repeated. When the object is not detected (that is, when the object is removed from the operation plane), the process proceeds to step S1108. Accordingly, the GUI component can accept the user's gesture operation, and the user's gesture operation is not ignored. Thereafter, this process is terminated.

  According to the processing in FIG. 11, when the power of the controller unit 201 is off and the power key 210 is pressed, the user's gesture operation on the GUI component is ignored as long as an object continues to exist on the operation plane. . Thereby, for example, even if the user activates the camera scanner 101 in a state where the user does not intend to operate the GUI part and puts the hand or pen at a position corresponding to the GUI part, the GUI part by the hand or pen is used. Operation on is ignored. Thereby, it is possible to prevent an erroneous operation on the GUI component by a hand or a pen placed unintentionally for the operation of the GUI component. Note that the processing of FIG. 11 can be applied not only when the camera scanner 101 is activated, but also when the camera scanner 101 returns from the sleep mode.

  Next, a third embodiment of the present invention will be described. Since the configuration and operation of the third embodiment are basically the same as those of the first embodiment described above, the description of the overlapping configuration and operation will be omitted, and different configurations and operations will be described below. Give an explanation.

  FIG. 12 is a flowchart showing projector turn-off / re-lighting processing as a control method of the display operation system according to the present embodiment. In the process of FIG. 12 as well, the user presses the turn-off key 211 to turn off the short focus projector 207 and then the user presses the turn-off key 211 again to the document table 204 in the same manner as the processes in FIGS. The main control unit 602 executes when projecting and displaying the display screen.

  First, when the user presses the turn-off key 211 while the display screen is projected and displayed from the short focus projector 207 to the document stage 204, the display control unit 613 notifies the main control unit 602 that the turn-off key 211 has been pressed. In response to the notification that the turn-off key 211 is pressed, the main control unit 602 executes main processing S908.

  Next, the main control unit 602 determines whether or not an object existing on the operation plane has changed between the time when the short focus projector 207 is turned off and the time when it is turned on again (step S909). If the result of determination in step S909 is that the object has not changed, the main control unit 602 executes step S910 and step S911. As a result, the GUI component can accept the user's gesture operation. Thereafter, this process is terminated.

  If the result of determination in step S909 is that the object has changed, the main control unit 602 executes step S912, and further performs object removal screen display processing that requests the user to remove the object present on the operation plane ( Step S1201). FIG. 13 is a diagram showing an example of the object removal screen displayed in step S1201 of FIG. As shown in FIG. 13, in step S1201, a message 1301 for requesting the user to remove the pen 804 or the hand 805 as an object is displayed on the operation plane of the document table 204. Note that the message 1301 may indicate that the user's gesture operation on the GUI component is ignored.

  Next, the main control unit 602 executes step S914, step S915, and step S913 in this order. In other words, unless the removal of the object existing on the operation plane is removed, the user's gesture operation on the GUI component is ignored even though the GUI component is projected and displayed on the operation plane of the document table 204. On the other hand, when the removal of the object existing on the operation plane is removed, the GUI component can accept the user's gesture operation, and the user's gesture operation is not ignored. Thereafter, this process is terminated.

  According to the processing of FIG. 12, when the short focus projector 207 is turned on again after the short focus projector 207 is turned off and the object is not removed from the operation plane, the message 1301 for requesting the user to remove the object is displayed. Is done. Thereby, the user can know the reason why the user's gesture operation with respect to the GUI component is ignored, and thus the convenience for the user can be maintained.

  In the processing of FIG. 12, the display of the message 1301 may be interrupted when the removal of the object existing on the operation plane is removed. Thereby, it can be known that the user's gesture operation with respect to the GUI component is possible, and the convenience for the user can be improved.

  Next, a fourth embodiment of the present invention will be described. Since the configuration and operation of the fourth embodiment are basically the same as those of the second embodiment described above, the description of the overlapping configuration and operation is omitted, and different configurations and operations are described below. Give an explanation.

  FIG. 14 is a flowchart showing a power-on process as a control method of the display operation system according to the present embodiment. The process in FIG. 14 is also executed by the main control unit 602 when the user presses the power key 210 in the controller unit 201 of the camera scanner 101.

  First, when the power of the controller unit 201 is off and the power key 210 is pressed, the main control unit 602 executes a startup process S1105. Next, the main control unit 602 determines whether an object present on the operation plane of the document table 204 is detected in step S1102 (step S1106). If the object is not detected as a result of the determination in step S1106, the main control unit 602 executes step S1107 and step S1108. As a result, the GUI component can accept the user's gesture operation. Thereafter, this process is terminated.

  If the object is detected as a result of the determination in step S1106, the main control unit 602 executes step S1109, and further performs the object removal screen display process in step S1201 in the process of FIG. Next, the main control unit 602 executes Step S1111, Step S1112, and Step S1110 in this order. In other words, unless the removal of the object existing on the operation plane is removed, the user's gesture operation on the GUI component is ignored even though the GUI component is projected and displayed on the operation plane of the document table 204. On the other hand, when the removal of the object existing on the operation plane is removed, the GUI component can accept the user's gesture operation, and the user's gesture operation is not ignored. Thereafter, this process is terminated.

  According to the processing of FIG. 14, when the power of the controller unit 201 is off and the power key 210 is pressed, and the object is not removed from the operation plane, the message 1301 that requests the user to remove the object. Is displayed. Thereby, the user can know the reason why the user's gesture operation with respect to the GUI component is ignored, and thus the convenience for the user can be maintained.

  Also in the process of FIG. 14, the display of the message 1301 may be interrupted when the removal of the object existing on the operation plane is removed, as in the process of FIG. 13. Thereby, it can be known that the user's gesture operation with respect to the GUI component is possible, and the convenience for the user can be improved.

  In the first to fourth embodiments described above, when an object is detected on the operation plane, the gesture operation on the GUI component is ignored. However, the user who stores the position of the GUI component is included in the GUI component. The associated process may be rushed to execute. In this case, it is conceivable that the user inserts a hand or a pen toward a desired GUI component in advance before the short-focus projector 207 that has been turned off is turned on again or before the camera scanner 101 is activated. Correspondingly, even if it is determined in step S909 that the object has changed, or it is determined in step S1106 that the object has been detected, the user's gesture for the GUI component under certain conditions It is preferable from the viewpoint of convenience not to ignore the operation. For example, even if it is determined in step S909 that the object has changed, if the detected object is an object that clearly intends to operate the GUI component, the user performs a gesture operation on the GUI component. Do not ignore. As a thing clearly intended to operate the GUI part, a hand moving toward the GUI part, a pen, a pointing stick (stylus), or the like is applicable.

  Next, a fifth embodiment of the present invention will be described. Since the configuration and operation of the fifth embodiment are basically the same as those of the first to fourth embodiments described above, the description of the overlapping configuration and operation is omitted, and different configurations are described below. The operation will be described. Specifically, in the first to fourth embodiments, even if a plurality of GUI parts are projected and displayed, gesture operations for all the GUI parts are ignored unless an object is removed from the operation plane. The On the other hand, in the fifth embodiment, when a plurality of GUI parts are projected and displayed, it is determined whether or not an object exists on each GUI part, and the GUI part on which an object exists. Only gesture operations on are ignored.

  FIG. 15 is a flowchart showing screen display processing as a control method of the display operation system according to the present embodiment. The processing in FIG. 15 is executed by the main control unit 602 when the short focus projector 207 is turned on again after the short focus projector 207 is turned off, or when the power source key 210 is pressed when the power of the controller unit 201 is off. Further, the processing in FIG. 15 is executed by the main control unit 602 that is notified of the execution of the screen transition from the application unit 614 even when the application unit 614 executes the screen transition in accordance with the user operation.

  First, the main control unit 602 executes object detection processing on the operation plane of the document table 204 by the object detection unit 609, and detects an object existing on the operation plane (step S1501). Thereafter, the main control unit 602 requests the GUI control unit 615 to display a plurality of GUI parts (step S1502). In the present embodiment, data of a plurality of GUI parts is transmitted from the application unit 614 to the GUI control unit 615.

  Next, the GUI control unit 615 performs the processing from step S1506 to step S1510 between step S1504 and step S1505 described below for each of the plurality of GUI components. That is, the processing from step S1506 to step S1510 is repeated for each GUI component.

  First, the main control unit 602 determines whether or not the object detected in step S1501 exists on the target GUI component for the target GUI component (hereinafter referred to as “target GUI component”) (step S1504). . If the result of determination in step S1504 is that the object does not exist on the target GUI component, execution of processing associated with the target GUI component is permitted (step S1506). If the process is not associated with the target GUI component, step S1506 is skipped. Thereafter, the main control unit 602 displays the target GUI component on the operation plane in the normal mode (step S1507). In the present embodiment, the “normal mode” indicates a mode in which the target GUI component is displayed in a state where the user's gesture operation can be accepted. As a result of the determination in step S1504, when the object exists on the target GUI component, the main control unit 602 determines whether or not the process is associated with the target GUI component (step S1508). If the result of determination in step S1508 is that processing is not associated with the target GUI component, processing proceeds to step S1507. If the process is associated with the target GUI component, execution of the process associated with the target GUI component is prohibited (step S1506). If execution of the process associated with the target GUI component is prohibited in step S1506, the associated process is naturally not performed even if the user operates the target GUI component with a gesture. That is, the user's gesture operation on the target GUI component is substantially ignored. Thereafter, the main control unit 602 displays the target GUI component on the operation plane in the operation prohibition mode (step S1507). In the present embodiment, the “operation prohibition mode” is a mode in which the target GUI component is displayed in a state where the user's gesture operation cannot be accepted. In the operation prohibition mode, as shown in FIG. 16, a button 802 that is a target GUI component for which execution of the associated process is prohibited because the pen 804 is placed is shaded and displayed. In the operation prohibition mode, the target GUI component for which execution of the associated process is prohibited may be displayed in a blinking manner. In FIG. 16, since there is no object on the button 803, the button 803 is displayed in the normal mode, and the user's gesture operation on the button 803 is not ignored. The user can know that the gesture operation on the target GUI component is ignored by confirming the shaded display or blinking display of the target GUI component. After that, when the main control unit 602 executes the processing from step S1506 to step S1510 for all the GUI components, this processing ends.

  FIG. 17 is a flowchart showing a GUI component operation permission process as a control method of the display operation system according to the present embodiment. The process of FIG. 17 is executed following the process of FIG.

  First, the main control unit 602 executes object detection processing on the operation plane of the document table 204 by the object detection unit 609, and detects an object existing on the operation plane (step S1501).

  Next, the GUI control unit 615 performs the processing from step S1704 to step S1706 between step S1702 and step S1703 for each GUI component for which execution of the processing associated in step S1509 in FIG. 12 is prohibited. That is, the processing from step S1704 to step S1706 is repeated for each GUI component for which execution of the associated processing is prohibited. The processing from step S1704 to step S1706 will be described below.

  First, the main control unit 602 detects the object detected in step S1701 on the prohibited GUI component for the target GUI component for which execution of associated processing is prohibited (hereinafter referred to as “prohibited GUI component”). It is determined whether or not to perform (step S1704). If the result of determination in step S1704 is that the object does not exist on the prohibited GUI component, execution of the process associated with the prohibited GUI component is permitted (step S1705). Thereafter, the main control unit 602 corrects the operation prohibited mode to the normal mode and displays the prohibited GUI component on the operation plane (step S1706). If the result of determination in step S1704 is that an object still exists on the prohibited GUI part, the operation prohibit mode is not corrected. Thereafter, when the main control unit 602 executes the processing from step S1704 to step S1706 for all the prohibited GUI components, it determines whether all the GUI components are displayed in the normal mode (step S1706). As a result of the determination in step S1706, if all the GUI parts are not displayed in the normal mode, the process returns to step S1701, and if all the GUI parts are displayed in the normal mode, this process is terminated.

  According to the processing of FIGS. 15 and 17, when the short focus projector 207 that has been turned off is turned on again or when the power key 210 is pressed when the camera scanner 101 is activated, the GUI component from which no object has been removed. The user's gesture operation on is ignored. That is, even if each GUI component is projected and displayed while the hand or pen is placed on a position corresponding to one of the plurality of GUI components unintentionally, the GUI component on which the hand or pen is placed is displayed. User gesture operations are ignored. Thereby, it is possible to prevent the occurrence of an erroneous operation on each of the plurality of GUI parts to be projected and displayed. Further, even if a hand or pen is placed on one of the plurality of GUI parts, if the hand or pen is not placed on another GUI part, the user's gesture operation on the other GUI part is not ignored. As a result, it is possible to ensure convenience related to the operation of other GUI components. Also in this embodiment, if there is even one GUI component from which an object has not been removed, a message 1301 requesting the user to remove the object may be displayed.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors of the computer of the system or apparatus read the program It can also be realized by processing executed in the above. The present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101 Camera scanner 201 Controller part 202 Camera part 204 Document stand 207 Short focus projector 208 Distance image sensor part 210 Power key 211 Off key 802, 803 button
1301 Message

Claims (13)

A display unit that projects and displays the operation parts of the application on the operation surface;
A detection unit for detecting an object present on the operation surface;
A display operation system including a control unit that controls the operation of the application based on an operation on the operation component,
When the display unit displays the operation component on the operation surface on which the operation component is not displayed, the control unit ignores an operation on the operation component when the detection unit detects the object. Characteristic display operation system.   When the detection unit detects the object when the display unit displays the operation component on the operation surface on which the operation component is not displayed, the display unit detects the object from above the operation surface. The display operation system according to claim 1, wherein a request for removal or an indication that an operation on the operation component is ignored is displayed.   When the detection unit detects removal of the object from the operation surface, the display unit ignores a request for removal of the object from the operation surface or an operation on the operation component. The display operation system according to claim 2, wherein display of the effect is stopped.   The display operation system according to claim 1, wherein the detection unit includes a camera and a three-dimensional distance sensor.   When the display unit displays the operation component again after interrupting the projection display of the operation component, an object detected after the projection display of the operation component again interrupts the projection display of the operation component. 5. The display operation system according to claim 1, wherein the control unit does not ignore the operation on the operation component when the object is not moved from the position of the object detected after the operation.   6. The control unit according to claim 1, wherein if the object detected by the detection unit is an object intended to operate the operation component, the control unit does not ignore the operation on the operation component. The display operation system described.   The display operation system according to claim 6, wherein the object intended to operate the operation component is a hand or a pointer that moves toward the operation component. A display unit that projects and displays a plurality of operation parts of the application on the operation surface;
A detection unit for detecting an object present on the operation surface;
A display operation system including a control unit that controls the operation of the application based on an operation on each of the plurality of operation components;
When the display unit displays the plurality of operation components on the operation surface on which the plurality of operation components are not displayed, the detection unit exists on the operation component for each of the plurality of operation components. When detecting the object, the control unit ignores an operation on the operation component.   When the display unit displays the plurality of operation components on the operation surface on which the plurality of operation components are not displayed, the detection unit exists on the operation component for each of the plurality of operation components. The said display part displays that the request | requirement of the removal from the said operation component of the said object or the operation with respect to the said operation component is disregarded when the said object is detected. Display operation system. A display step for projecting and displaying the operation parts of the application on the operation surface;
A detection step of detecting an object present on the operation surface;
And a control step of ignoring an operation on the operation component when the object is detected when the operation component is displayed on the operation surface on which the operation component is not displayed. How to control the system. A display step for projecting and displaying a plurality of operation parts of the application on the operation surface;
A detection step of detecting an object present on the operation surface;
In the case where the plurality of operation components are displayed on the operation surface on which the plurality of operation components are not displayed, the object existing on the operation component in the detection step is related to each of the plurality of operation components. And a control step of ignoring an operation on the operation component when detected. A program for causing a computer to execute a control method of a display operation system,
The display operation system control method includes:
A display step for projecting and displaying the operation parts of the application on the operation surface;
A detection step of detecting an object present on the operation surface;
And a control step of ignoring an operation on the operation component when the object is detected when the operation component is displayed on the operation surface on which the operation component is not displayed. A program for causing a computer to execute a control method of a display operation system,
A display step for projecting and displaying a plurality of operation parts of the application on the operation surface;
A detection step of detecting an object present on the operation surface;
In the case where the plurality of operation components are displayed on the operation surface on which the plurality of operation components are not displayed, the object existing on the operation component in the detection step is related to each of the plurality of operation components. And a control step of ignoring an operation on the operation component when detected.

Images