JP2006252037A - Specifying operation detector, program, specifying operation detecting method, and operation detecting system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily and accurately detect the existence or nonexistence of specifying the operation of a user with respect to a displayed image, and a position to which the specifying operation has been carried out. <P>SOLUTION: The specifying operation detector 8 is provided with an operation storing means 15 for storing operation receiving position data indicating a position for receiving specifying operation from a user in a display area of an image output device 2, an image creating means 17 for providing image output data 9 prompting specifying operation to the user to the image output device 2, a specified position detecting means for creating specified position data indicating a specified position by the user on the basis of the image input data 10 of the display area inputted by the image input device 4, an operation sound detecting means 31 for detecting a specifying operation sound generated by the user on the basis of sound input data 11 measured by a sound input device 6, and an operation judging means 16 for determining that specifying operation responding to the position indicated by the operation receiving position data has been carried out by the user when the specifying operation sound is detected, and the operation receiving position data matches with the specified position data. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a designated operation detection device, a program, a designated operation detection method, and an operation detection system that detect the presence or absence of a designated operation by a user.

  Non-Patent Document 1 (Hiroshi Ishii, Craiq Wisneski, Julian Orbanes, Ben Chun, Joe Paradiso, “PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play”, Proceedings of the SIGCHI conference on Human factors in computing systems, (May 1999, pp.394-401) discloses a technique for measuring sound and detecting the position of the sound source. In Non-Patent Document 1, four microphones are installed in order to detect the position where the ping-pong ball has dropped.

  Non-Patent Document 2 (Yuji Ayatsuka, Junichi Kyokumoto, “Real-world application using a simple knock position detection device” in WISS2001, Dec.2001, Interactive System and Software IX, Modern Science pp.191-196) Has proposed a method of detecting a knock position with respect to a wall or a desk using a vibration sensor.

Hiroshi Ishii, Craiq Wisneski, Julian Orbanes, Ben Chun, Joe Paradiso, “PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play”, Proceedings of the SIGCHI conference on Human factors in computing systems, May 1999, pp.394-401

Yuji Ayatsuka, Junichi Kyokumoto, “Real-world application using a simple knock position detector” in WISS2001, Dec.2001, Interactive System and Software IX, Modern Science Co., Ltd. pp.191-196

  The non-patent document 1 detects a falling position of a ping-pong ball and does not detect the presence or absence of a button operation. Further, in this Non-Patent Document 1, it is necessary to install microphones at four locations, and it is necessary to adjust and calibrate the apparatus.

  In Non-Patent Document 2, three knocks are detected in order to prevent malfunction due to vibration, reflected sound, etc. caused by factors other than knock. Therefore, it is difficult to use the technique of Non-Patent Document 2 for detecting, for example, one pressing operation of the user on the displayed button image.

  The present invention has been made in view of the above circumstances, and a designated operation detection device that easily and accurately detects the presence / absence of a user's designated operation on a displayed image and the position where the designated operation has been performed, It is an object to provide a program, a designated operation detection method, and an operation detection system.

  In the aspect of the present invention, the designation operation detection device connected to the image output device, the image input device, and the sound input device has operation reception position data representing a position where the designation operation is received from the user in the display area of the image output device. Based on operation storing means for storing, image output data for prompting a user to perform a designation operation, image creating means for providing the image output data to the image output apparatus, and image input data of the display area input by the image input apparatus A designated position detecting means for creating designated position data indicating a designated position of the user, and an operation sound detecting means for detecting a designated operation sound generated by the user based on the sound input data measured by the sound input device; When the specified operation sound is detected and the operation reception position data matches the specified position data, the operation reception position is determined by the user. Comprising an operation determining means for determining a designated operation corresponding to the position indicated by the data has been executed.

Thereby, the presence / absence of the user's designation operation is detected from the designation operation sound, and the position where the designation operation is performed by the user is detected from the photographed image.
As a result, it is possible to easily and accurately detect the presence / absence of the designation operation and the position where the designation operation is performed by at least one designation operation by the user.

In the specified operation detection device, the image output device may be a projector.
In the specified operation detection device, the image output data is data indicating an operation button, the operation reception position data is data indicating a display position of the operation button, and the image input data is captured by the image input device. This is data indicating an image around the display position of the operation button, the specified position data is data indicating the result of detecting the position of the user's hand region, and the specified operation sound is the data of the operation button displayed by the image output device. The contact sound between the display surface and the user's hand, and the operation determining means detects the contact sound between the display surface of the operation button and the user's hand, and the position of the user's hand region and the display position of the operation button are If they overlap, it may be determined that the designation operation for the operation button has been executed.

  In the above-described designation operation detection device, the operation storage means stores the operation reception position data and the output data in association with each other, and the image creation means designates according to the position indicated by the operation reception position data by the operation determination means. When it is determined that the operation has been executed, output of the output data may be executed. Further, in the designation operation detecting device in this case, the image creating means determines the data type of the output data, and when the output data is image data, creates image output data including the output data, and includes an image including the output data. Sound output means may be further provided for providing the output data to the image output device and providing the output data to the sound output device when the output data is sound data.

In the specified operation detecting device, the operation sound detecting means calculates an average value of the sound input data every predetermined period, and when the average value is larger than the operation sound detection threshold, the specified operation sound is generated by the user. It may be detected.
In addition, said aspect is expressed as an apparatus. However, the present invention is not limited to this, and the above aspect is expressed by a method, a program, a computer-readable storage medium, a system including an image output apparatus, an image input apparatus, a sound input apparatus, and the apparatus of the above aspect. Also good.

  According to the present invention, it is possible to easily and accurately detect the presence / absence of a user's designation operation on the displayed image and the position where the designation operation has been performed.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, the same code | symbol is attached | subjected about the same part in each following figure, and description is abbreviate | omitted.
FIG. 1 is a block diagram showing an example of the configuration of the operation detection system according to the present embodiment.
FIG. 2 is a diagram illustrating an application example of the operation detection system according to the present embodiment.

  In this embodiment, the image output device is described as a projector, the image input device as a camera, the sound input device as a microphone, and the sound output device as a speaker. Note that the image output device may be another display device such as a CRT or a liquid crystal display. In the present embodiment, the display range by the image output device and the shooting range by the image input device are matched.

  The operation detection system 1 according to the present embodiment first projects the operation button 3 onto the real space using the projector 2, and the projected area of the operation button 3 is photographed by the camera 4 and the projected operation is performed. Based on the combination of the sound recognition result when the button 3 is struck and the image recognition result for determining whether or not the hand 5 is over the operation button area, the operation button 3 has been pressed. Detecting and performing device operation and processing in response to pressing of the operation button 3.

The operation detection system 1 includes a projector 2, a camera 4, a microphone 6, a speaker 7, and a designated operation detection device 8.
The projector 2 receives the image output data 9 created by the designation operation detecting device 8, projects an image on the real space based on the image output data 9, and displays the image.

The camera 4 shoots the real space state such as the user's instruction state and operation state, the state of the image projected by the projector 2 in the real space, and designates the image input data 10 indicating the real space state to detect the designated operation. Pass to device 8.
In the present embodiment, the projection position of the projector 2 and the shooting position of the camera 4 are matched.

The microphone 6 inputs a designated operation sound that is emitted when the user performs a designated operation, and passes the sound input data 11 to the designated operation detection device 8.
The speaker 7 receives sound output data 12 corresponding to the user's designated operation from the designated operation detection device 8 and outputs sound according to the sound output data 12.

The designated operation detection device 8 includes an image recognition unit 13, a sound recognition unit 14, an operation database 15, an operation determination unit 16, an image creation unit 17, a sound output data storage unit 18, and a sound provision unit 19.
The image recognition unit 13 includes an image acquisition unit 20, an image input data storage unit 21, a hand region detection unit 22, a hand region data storage unit 23, a centroid calculation unit 24, and a centroid data storage unit 25. Among these, the hand region detection unit 22, the hand region data storage unit 23, the centroid calculation unit 24, and the centroid data storage unit 25 constitute a designated position detection unit 26.

Note that the image recognition unit 13 may detect not only the detection of the hand region but also various user-designated positions such as the tip of a pointing bar and the pointing position of a pointer device.
The image acquisition unit 20 acquires the image input data 10 in the space around the operation button 3 projected by the projector 2 from the camera 4 and stores the image input data 10 in the image input data storage unit 21.

  The hand region detection unit 22 reads the image input data 10 from the image input data storage unit 21, and detects a skin color region indicating the worker's hand 5 as a hand region by image analysis on the image input data 10. Then, the hand region detection unit 22 creates hand region data 27 indicating the hand region detection result and stores it in the hand region data storage unit 23.

  Note that even when the specified operation is not performed by the hand 5, it is only necessary to detect an object that contacts the projected operation button 3. When an operator wears gloves, this glove is detected. For example, when the operation button 3 is designated by the tip of a laser pointer or a pointer, the irradiation position of the laser pointer or the tip of the pointer is detected.

  The center-of-gravity calculation unit 24 reads the hand region data 27 from the hand region data storage unit 23, calculates the position (coordinate value) of the center of gravity of the hand region in the hand region data 27, and creates the center of gravity data 28 indicating the position of the center of gravity. And stored in the gravity center data storage unit 25. Then, the center-of-gravity calculation unit 24 issues a hand region detection event Ecap including the center-of-gravity data 28.

The sound recognition unit 14 includes a sound acquisition unit 29, a sound input data storage unit 30, and an operation sound detection unit 31.
The sound acquisition unit 29 acquires the sound input data 11 about the space around the operation button 3 projected by the projector 2 from the microphone 6 and stores the sound input data 11 in the sound input data storage unit 30.

  The operation sound detection unit 31 reads the sound input data 11 from the sound input data storage unit 30 and detects a user-specified operation sound (for example, a knock sound) by sound analysis such as amplitude analysis on the sound input data 11. And the operation sound detection part 31 emits the operation sound detection event Em, when a designated operation sound is detected.

The operation database 15 associates, for each operation button, operation button ID, button name, shape data, size data, operation button display position data (X coordinate value, Y coordinate value) corresponding to the operation reception position, and attribute data. The operation button data 32 is stored.
The operation database 15 stores output data 331 to 33N corresponding to the operation buttons.

FIG. 3 is a diagram illustrating an example of the contents stored in the operation database 15.
In the operation button data 32, the operation button ID, button name, shape data, size data, display position data, and attribute data are managed in a table format for each of the plurality of operation buttons. The size data is represented by a radius when the shape data indicates a circle, and by a radius of a circumscribed circle when the shape data indicates a square or a triangle.

As the display position data, an X-coordinate value and a Y-coordinate value indicating the position where each operation button is displayed are managed.
The attribute data includes a path to output data 331 to 33N that is output (for example, displayed as an image or output as sound) when each operation button is pressed (path to image data, path to moving image data, sound Data path, text data path, etc.) are stored. As attribute data, for example, notes, features, operation methods, and next operations corresponding to the operation buttons may be stored. The attribute data itself may be output data 331 to 33N. The attribute data may be a path or link information to a program or script.

The operation determination unit 16 reads various data such as the operation button ID, position data, shape data, and size data of the operation button 3 from the operation database 15.
Further, when the operation determination unit 16 receives the hand region detection event Ecap including the centroid data 28 and the operation sound detection event Em, the position of the centroid of the hand region indicated by the centroid data 28 is within the projection range of the operation button 3. If the position of the center of gravity of the hand area is included in the projection range of the operation button 3, an operation generation event is generated and the operation button ID of the operation button 3 in which the pressing operation has occurred is determined as an image creation unit. 17 to provide.

The image creating unit 17 includes a response data creating unit 34, an image output data storage unit 35, and an image providing unit 36.
The response data creation unit 34 includes an image output data creation function 34a, a data type determination function 34b, and a sound output data creation function 34c.
The image output data creation function 34 a of the response data creation unit 34 accesses the operation database 15, creates image output data 9 for displaying the operation button 3 at the position indicated by the display position data, and stores it in the image output data storage unit 35. To do.

  The data type determination unit 34b of the response data creation unit 34 receives the operation button ID from the operation database 15 when an operation occurrence event occurs and the operation button ID of the operation button 3 that has been pressed is received from the operation determination unit 16. The output data corresponding to the ID is read, and the data type of this output data is determined.

  When the output data is image data or moving image data, the image output data creating function 34a of the response data creating unit 34 displays the operation button 3 and image output data 9 for displaying image data or moving image data as output data. Is stored in the image output data storage unit 35.

When the output data is sound data, the sound output data creation function 34 c of the response data creation unit 34 stores the sound data as the output data in the sound output data storage unit 18 as the sound output data 12.
The image providing unit 36 provides the image output data 9 stored in the image output data storage unit 35 to the projector 2.

The sound providing unit 19 provides the speaker 7 with the sound output data 12 stored in the sound output data storage unit 18.
The operation detection system 1 as described above inputs sound around the operation button 3 projected on the projection plane of the space by the projector 2 with the microphone 6 and also displays an image around the operation button 3 projected on the projection plane with the camera 4. Shoot with.

  And the operation detection system 1 detects the sound emitted when the user's hand 5 contacts the operation button 3 projected on the projection surface, and the projection position of the operation button 3 and the user's hand 5 overlap. When the state is detected, it is determined that the user has designated the operation button 3.

When a designation operation by the user for the operation button 3 is performed, the operation detection system 1 determines the data type of the output data corresponding to the operation button 3 that has been designated.
When the output data is image data, the operation detection system 1 projects the contents of the image data with the projector 2, and when the output data is sound data, the contents of the sound data are output from the speaker 7.

  As described above, the operation detection system 1 according to the present embodiment includes the projector 2, the camera 4, the microphone 6, the speaker 7, and the designated operation detection device 8. The designated operation detection device 8 performs processing such as image recognition and voice recognition. A computer may be used as the designated operation detection device 8. The projection range of the projector 2 and the shooting range of the camera 4 are matched. The operation button 3 to be projected and various data (for example, text data, sound data, image data, processing script, etc.) associated with the operation button 3 are managed by the designated operation detection device 8. The operation button 3 is projected on the projection surface by the projector 2. When the projected operation button 3 is pressed by the user, various data associated with the operation button 3 are presented to the user.

FIG. 4 is a flowchart showing an example of processing of the image recognition unit 13 of the designation operation detection device 8 according to the present embodiment.
In step S1, the image acquisition unit 20 sets an image acquisition period Tcap.
In step S <b> 2, the image acquisition unit 20 acquires the image input data 10 from the camera 4 and stores it in the image input data storage unit 21.

  In step S <b> 3, the hand region detection unit 22 performs image analysis on the image input data 10 in the image input data storage unit 21, detects a skin color region indicating the worker's hand 5 as a hand region, and detects a hand region. The hand region data 27 indicating the result is stored in the hand region data storage unit 23.

In step S <b> 4, the center-of-gravity calculation unit 24 determines whether there is a hand region for the hand region data 27 in the hand region data storage unit 23.
If there is no hand area, in step S5, the image recognition unit 13 waits until the image acquisition cycle Tcap elapses, and then the process returns to step S2.

On the other hand, when there is a hand area, in step S6, the center-of-gravity calculation unit 24 calculates the position of the center of gravity of the hand area with respect to the hand area data 27 of the hand area data storage unit 23, and obtains the center-of-gravity data 28 indicating the position of the center of gravity. Stored in the centroid data storage unit 25.
In step S7, the center-of-gravity calculation unit 24 generates a hand region detection event Ecap having as an argument the coordinate value of the hand region center of gravity indicating the position of the center of gravity, and the process proceeds to step S5.

FIG. 5 is a flowchart showing an example of processing of the sound recognition unit 14 of the designation operation detection device 8 according to the present embodiment.
In step T1, the operation sound detector 31 sets an operation sound detection threshold Mt.
In step T2, the sound acquisition unit 29 sets an average value calculation cycle Tm.

Note that the execution order of steps T1 and T2 may be reversed or may be executed in parallel.
In step T <b> 3, the sound acquisition unit 29 acquires the sound input data 11 during the average value calculation period Tm from the microphone 6 for the space around the operation button 3, and stores the sound input data 11 in the sound input data storage unit 30. To do.

In step T4, the operation sound detection unit 31 calculates the average value m of the sound input data 11 during the average value calculation cycle Tm in the sound input data storage unit 30.
In step T5, the operation sound detection unit 31 determines whether or not the average value m of the sound input data 11 during the average value calculation period Tm is greater than the operation sound detection threshold Mt.

If the average value m is less than or equal to the operation sound detection threshold Mt, the process returns to step T3.
When the average value m is larger than the operation sound detection threshold Mt, in step T6, the operation sound detection unit 31 generates an operation sound detection event Em, and then the process returns to step T3.

FIG. 6 is a flowchart showing an example of processing of the operation determination unit 16 of the designated operation detection device 8 according to the present embodiment.
In step U <b> 1, the operation determination unit 16 acquires display position data, shape data, and size data of the operation button 3 from the operation database 15.

In step U <b> 2, the operation determination unit 16 determines whether or not the operation sound detection event Em has occurred in the operation sound detection unit 31.
If the operation sound detection event Em has not occurred, the process repeats step U2 and enters a state of waiting for the operation sound detection event Em to be generated.

When the operation sound detection event Em occurs, in step U3, the operation determination unit 16 determines whether or not the hand region detection event Ecap has occurred in the gravity center calculation unit 24.
If the hand region detection event Ecap has not occurred, the process returns to step U2.

The steps U2 and U3 may be reversed or executed in parallel.
When the hand region detection event Ecap occurs, in step U4, the operation determination unit 16 determines whether the position (coordinate value) of the center of gravity of the hand region indicated by the center of gravity data 28 is within the projection range of the operation button 3.

If the position of the center of gravity of the hand region is not within the projection range of the operation button 3, the process returns to step U2.
If the position of the center of gravity of the hand region is within the projection range of the operation button 3, in step U5, the operation determining unit 16 operates the operation button in a state where the position of the center of gravity of the hand region is included in the projection range of the operation button 3. The ID and the operation occurrence event are issued to the image creating unit 17, and then the process returns to Step U2. The operation button ID may be included in the operation occurrence event.

  FIG. 7 is a flowchart illustrating an example of processing performed by the image creating unit 17 and the sound providing unit 19 of the designation operation detecting device 8 according to the present embodiment. In FIG. 7, the process for the operation button 3 with the operation button ID k is described, but the same applies to the operation buttons with other operation button IDs. Further, in FIG. 7, a case where output data is output will be described as an example, but the same applies to a case where attribute data or the like is output, for example.

In FIG. 7, when the operation button 3 projected on the projection surface is pressed, output data corresponding to the operation button 3 is output. When the operation button 3 is pressed again, the output corresponding to the operation button 3 is output. Data output is stopped.
In step V1, the response data creation unit 34 indicates a state in which output data is not provided for the output data provision flag Flag (k) indicating whether or not to provide output data for the operation button 3 having the operation button ID k. Set False.

In step V <b> 2, the response data creation unit 34 acquires display position data, shape data, and size data of the operation button 3 from the operation database 15.
In step V <b> 3, the response data creating unit 34 displays image output data for displaying the operation button 3 corresponding to the shape data and the size data at a position corresponding to the display position data based on the data acquired from the operation database 15. 9 is prepared and provided to the projector 2 via the image output data storage unit 35 and the image providing unit 36. Thereby, the operation button 3 is displayed on the projection surface.

In step V <b> 4, the response data creation unit 34 determines whether an operation occurrence event has occurred from the operation determination unit 16.
If an operation occurrence event has not occurred, the process repeats step V4 and enters an operation occurrence event waiting state.
When an operation occurrence event has occurred, in step V5, the response data creation unit 34 acquires output data related to the operation button ID = k received from the operation determination unit 16 from the operation database 15.

In step V6, the response data creation unit 34 determines the type of output data.
When the output data is image data or moving image data, in step V7, the response data creation unit 34 determines whether False is set in the output data provision flag Flag (k).

  When False is set in the output data provision flag Flag (k), in step V8, the response data creation unit 34 creates the image output data 9 for displaying the operation button 3 and the output data, and outputs the image. The data is provided to the projector 2 via the data storage unit 35 and the image providing unit 36, the output data providing flag Flag (k) is set to true, and then the process returns to step V4.

  When the output data provision flag Flag (k) is set to True, in step V9, the response data creation unit 34 hides the displayed output data and outputs an image for displaying the operation button 3. Data 9 is created and provided to the projector 2 via the image output data storage unit 35 and the image providing unit 36, the output data providing flag Flag (k) is set to false, and then the process returns to step V4.

When the output data is sound data, in step V10, the response data creation unit 34 determines whether False is set in the output data provision flag Flag (k).
When the output data provision flag Flag (k) is set to False, in step V11, the response data creation unit 34 provides the output data to the speaker 7 via the sound output data storage unit 18 and the sound provision unit 19. The output data provision flag Flag (k) is set to true, and then the process returns to step V4.

  When the output data provision flag Flag (k) is set to True, in step V12, the response data creation unit 34 stops outputting the output data being reproduced, and sets the output data provision flag Flag (k) to false. After that, the process returns to step V4.

Note that. When the data type of the output data is text data, the content of the text data may be displayed or may be converted into sound and output.
FIG. 8 is a flowchart illustrating a processing example of the designation operation detection method according to the present embodiment.
In step W1, the response data creation unit 34 accesses the operation database 15 and creates image output data 9 for displaying the operation button 3 at the position indicated by the display position data.

In step W <b> 2, the image providing unit 36 provides the image output data 9 to the projector 2.
In step W <b> 3, the image acquisition unit 20 acquires the image input data 10 captured by the camera 4.
In step W4, the hand region detection unit 22 detects a hand region in the image input data 10 by a technique such as skin color detection.

If no hand area is detected, the process returns to step W3.
If a hand area is detected, the process proceeds to step W5.
In step W5, the center of gravity calculation unit 24 calculates the position (coordinate value) of the center of gravity H of the hand region, and the process proceeds to step W8.
In step W <b> 6, the sound acquisition unit 29 acquires the sound input data 11 around the projection position of the operation button 3 using the microphone 6.

In step W <b> 7, the operation sound detection unit 31 detects a user-specified operation sound by sound analysis on the sound input data 11.
If the designated operation sound is not detected, the process returns to step 6.
If the designated operation sound is detected, the process proceeds to step W8.
In step W <b> 8, the operation determination unit 16 determines whether the hand region is detected by the gravity center calculation unit 24 and whether the designated operation sound is detected by the operation sound detection unit 31.

If the hand area and the designated operation sound are not detected, the process returns to step W3 and step W6.
When the hand region and the designated operation sound are detected, in step W9, the operation determination unit 16 determines whether the position of the center of gravity of the hand region is within the projection range of the operation button 3.
If the position of the center of gravity of the hand region is not within the projection range of the operation button 3, the process returns to step W3 and step W6.

When the position of the center of gravity of the hand region is within the projection range of the operation button 3, the response data creation unit 34 determines the data type of the output data corresponding to the operation button 3 in the operation database 15 in step W10.
When the output data is image data or moving image data, in step W11, the response data creation unit 34 determines whether or not the image for the output data is in a display state.

In the state where the image for the output data is not displayed, in step W12, the response data creating unit 34 creates the image output data 9 for displaying the output data and the operation buttons 3.
When the image for the output data is in the display state, in step W13, the response data creation unit 34 creates the image output data 9 for stopping the display of the output data and displaying the operation button 3.

In step W14, the image providing unit 36 provides the image output data 9 to the projector 2, and the process returns to step W3 and step W6.
When the output data is sound data, in step W15, the response data creation unit 34 determines whether or not the sound for the output data is in an output state.

In the state where the sound for the output data is not output, in step W16, the sound providing unit 19 provides the sound output data 12 to the speaker 7, and the process returns to step W3 and step W6.
If the sound for the output data is in the output state, in step W17, the response data creation unit 34 stops outputting the output data, and the process returns to step W3 and step W6.

The example of the utilization aspect of the operation detection system 1 which concerns on this Embodiment is demonstrated.
For example, in a work site such as a plant, when the operation button 3 is projected only by setting the contents of the operation database 15 without installing an actual operation button, a designated operation is performed on the projected operation button 3. The data corresponding to the operation button 3 can be output. Accordingly, plant maintenance and information provision can be easily realized. In addition, by using the operation detection system 1, it is possible to output a handling method of equipment, caution items, an operation manual, and the like according to the contents of the operation database 15, and it is possible to easily cope with a change in output state.

Further, the operation database 15 can be updated at the support center, and the contents of the operation database 15 created at the support center can be quickly reflected on the work site. As a result, cooperative work between the site and the support center can be realized.
In the present embodiment described above, the microphone 6 is used to detect the presence / absence of a user-designated operation based on the intensity of sound, and image analysis is used to detect the operation position. Therefore, special calibration and adjustment of the apparatus at the time of introduction are unnecessary, and the introduction is easy.

  Further, in the present embodiment, by combining the sound recognition processing and the image analysis processing, it is possible to distinguish between the sound emitted when the user performs the designation operation and the other sound to improve accuracy, and to project the sound. It is possible to easily and accurately detect the presence / absence of the user's designation operation on the displayed image and the position where the designation operation has been performed.

In the present embodiment, the presence / absence of the designation operation and the designated content can be detected by a single designation operation by the user, and processing according to the designation operation can be executed.
In the present embodiment, the arrangement state of the operation buttons 3 and the contents of data provided when the operation buttons 3 are pressed can be easily changed / updated / added, and installation labor and cost can be greatly reduced.

  In this embodiment, each component may be rearranged as long as the same operation can be realized, each component may be freely combined, and each component may be freely divided. Alternatively, some components may be deleted. That is, the present embodiment is not limited to the above-described configuration as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage.

For example, as described above, not only the detection of the user's pressing operation on the projected operation button 3 but also the designation operation by the user by designating the position with a laser pointer or the tip of a pointing bar and emitting a designation operation sound. May be accepted.
In the present embodiment, the operation button 3 is displayed. For example, a schematic configuration diagram of a plant, a map in a factory, a region-divided image, a color-coded image, various symbols or marks, etc. Various images for prompting a user's specifying operation may be displayed, and detailed information corresponding to the position specified by the user may be displayed. In this case, the operation database 15 stores operation reception position data for receiving a designation operation from the user, instead of the display position data of the operation button 3 described above.

Further, for example, in the present embodiment, the operation sound detection unit 31 detects the contact sound between the projection surface of the operation button 3 and the user's hand 5 as the designated operation sound, but the electronic sound emitted by the user, etc. May be detected as the designated operation sound.
Further, for example, the storage units 21, 23, 25, 30, 35, 18 and the operation database 15 can be freely combined. The data stored in the storage units 21, 23, 25, 30, 35, 18 and the operation database 15 can be freely combined.

FIG. 9 is a block diagram showing an example of a configuration for realizing the designated operation detection device 8 of the operation detection system 1 according to the present embodiment using a computer and software.
The computer 37 includes a processor 38, a storage device 39 such as an internal memory, and an internal bus 40. The processor 38 and the storage device 39 are connected via an internal bus 40.

  The program 42 stored in the storage medium 41 is read into the computer 37 and executed by the processor 38. As a result, the computer 37 realizes the function as each constituent element of the designated operation detection device 8. The storage device 39 functions as the storage units 21, 23, 25, 30, 35, 18 and the operation database 15.

  In the present embodiment, the program 42 is written in a storage medium 41 such as a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, etc., and operates as the designated operation detection device 8. The present invention can be applied to the computer 37. The program 42 can also be applied to the computer 37 by being transmitted through a communication medium. The computer 37 reads the program 42 and controls the operation of the program 42, thereby realizing the function as the designated operation detection device 8. The program 42 may be arranged in a distributed manner on a plurality of computers, and the processing may be executed in cooperation with each other between the plurality of computers.

  The present invention is effective in the field of presenting information using a projector.

The block diagram which shows the example of a structure of the operation detection system which concerns on embodiment of this invention. The figure which shows the example of application of the operation detection system which concerns on the same embodiment. The figure which shows the example of the memory content of an operation database. The flowchart which shows the example of a process of the image recognition part of the designation | designated operation detection apparatus which concerns on the embodiment. The flowchart which shows the example of a process of the sound recognition part of the designation | designated operation detection apparatus which concerns on the same embodiment. The flowchart which shows the example of a process of the operation determination part of the designation | designated operation detection apparatus which concerns on the same embodiment. The flowchart which shows the example of a process with the image preparation part and sound provision part of the designation | designated operation detection apparatus which concerns on the embodiment. The flowchart which shows the process example of the designation | designated operation detection method which concerns on the embodiment. The block diagram which shows the example of the structure for implement | achieving the designation | designated operation detection apparatus of the operation detection system which concerns on the embodiment using a computer and software.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Operation detection system, 2 ... Projector, 3 ... Operation button, 4 ... Camera, 5 ... Hand, 6 ... Microphone, 7 ... Speaker, 8 ... Designated operation detection apparatus, 9 ... Image output data, 10 ... Image input data, DESCRIPTION OF SYMBOLS 11 ... Sound input data, 12 ... Sound output data, 13 ... Image recognition part, 14 ... Sound recognition part, 15 ... Operation database, 16 ... Operation judgment part, 17 ... Image preparation part, 18 ... Sound output data storage part, 19 DESCRIPTION OF SYMBOLS ... Sound providing unit, 20 ... Image acquisition unit, 21 ... Image input data storage unit, 22 ... Hand region detection unit, 23 ... Hand region data storage unit, 24 ... Center of gravity calculation unit, 25 ... Center of gravity data storage unit, 26 ... Designation Position detection unit, 27 ... hand region data, 28 ... centroid data, 29 ... sound acquisition unit, 30 ... sound input data storage unit, 31 ... operation sound detection unit, 32 ... operation button data, 331-33N ... output data, 34 ... response data 34a ... data type determination function, 34c ... sound output data creation function, 35 ... image output data storage unit, 36 ... image providing unit, 37 ... computer, 38 ... processor, 39 ... Storage device 40 ... Internal bus 41 ... Storage medium 42 ... Program

Claims (9)

In the designated operation detection device connected to the image output device, the image input device, and the sound input device,
Operation storage means for storing operation reception position data representing a position for receiving a designation operation from a user in a display area of the image output device;
Creating image output data for prompting the user to perform a designation operation, and providing the image output data to the image output device;
Designated position detection means for creating designated position data indicating the designated position of the user based on the image input data of the display area input by the image input device;
Operation sound detection means for detecting a designated operation sound generated by the user based on sound input data measured by the sound input device;
When the designated operation sound is detected and the operation reception position data matches the designated position data, an operation for determining that the designation operation corresponding to the position indicated by the operation reception position data has been executed by the user A designated operation detection device comprising: a determination unit; In the designation | designated operation detection apparatus of Claim 1,
The designation operation detection device, wherein the image output device is a projector. In the designation | designated operation detection apparatus of Claim 1 or Claim 2,
The image output data is data for displaying an operation button,
The operation reception position data is data indicating a display position of the operation button,
The image input data is data indicating an image around the display position of the operation button photographed by the image input device,
The designated position data is data indicating a result of detecting the position of the user's hand area;
The designated operation sound is a contact sound between the display surface of the operation button displayed by the image output device and the user's hand,
The operation determining means detects a contact sound between the display surface of the operation button and the user's hand, and when the position of the user's hand region and the display position of the operation button overlap, A designation operation detecting device characterized in that it is determined that a designation operation for a button has been executed. In the designation | designated operation detection apparatus of any one of Claim 1 thru | or 3,
The operation storage means stores the operation reception position data and output data in association with each other,
The image creating means executes the output of the output data when the operation determining means determines that the specifying operation corresponding to the position indicated by the operation reception position data has been executed. apparatus. In the designation | designated operation detection apparatus of Claim 4,
The image creation means determines the data type of the output data, and creates image output data including the output data when the output data is image data, and outputs the image output data including the output data to the image output Providing to the device,
The designated operation detecting device further comprising sound providing means for providing the output data to a sound output device when the output data is sound data. In the designation | designated operation detection apparatus of any one of Claim 1 thru | or 5,
The operation sound detection means calculates an average value of sound input data for each predetermined period, and detects that the specified operation sound is generated by the user when the average value is larger than the operation sound detection threshold value. A designated operation detecting device characterized in that: Computer
Operation storage means for storing operation reception position data representing a position for receiving a designation operation from the user in the display area of the image output device;
Image creation means for creating image output data for prompting the user to perform a designation operation, and providing the image output data to the image output device;
Designated position detection means for creating designated position data indicating the designated position of the user based on image input data of the display area input by an image input device;
An operation sound detecting means for detecting a designated operation sound generated by the user based on sound input data measured by the sound input device;
When the designated operation sound is detected and the operation reception position data matches the designated position data, an operation for determining that the designation operation corresponding to the position indicated by the operation reception position data has been executed by the user A program for functioning as a determination means. In a designated operation detection method using an image output device, an image input device, and a sound input device,
Storing operation reception position data representing a position for receiving a designation operation from a user in a display area of the image output device in an operation storage unit;
Creating image output data that prompts the user to specify, providing the image output data to the image output device;
Based on the image input data of the display area input by the image input device, creating specified position data indicating the specified position of the user, and based on the sound input data measured by the sound input device, Detect specified operation sound generated by the user,
When the designated operation sound is detected and the operation reception position data matches the designated position data, it is determined that a designation operation corresponding to the position indicated by the operation reception position data has been executed by the user. Designated operation detection method characterized by An image output device;
An image input device for photographing a display area of the image output device;
A sound input device for measuring sound around a display area of the image output device;
Operation storage means for storing operation reception position data representing a position for receiving a designation operation from a user in a display area of the image output device;
Creating image output data for prompting the user to perform a designation operation, and providing the image output data to the image output device;
Designated position detection means for creating designated position data indicating the designated position of the user based on image input data of the display area photographed by the image input device;
Operation sound detection means for detecting a designated operation sound generated by the user based on sound input data measured by the sound input device;
When the designated operation sound is detected and the operation reception position data matches the designated position data, an operation for determining that the designation operation corresponding to the position indicated by the operation reception position data has been executed by the user An operation detection system comprising a determination unit.

Images