JP2005300971A - System and method for supporting operation - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an operation support system with which guide information for supporting operations can be projected from a projector when a user operates or learns a complicated device, a musical instrument, etc. <P>SOLUTION: The operation support system is provided which supports an operation procedure that the user follows by using an operating means, and a camera (imaging means) 3 images in advance a nearby area including the operating means (keyboard) of a piano 1. Then the projector (projecting means) 2 detects a position of the operating means (keyboard) that the user should operate from image information picked up by the camera (imaging means) 3 and projects given information (for example, light spot) on the detected position of the operating means (keyboard). At this time, a PC 5 controls positions of the projector (projecting means) 2 and camera (imaging means) 3 based upon musical score information recorded on a CD-ROM 4, so the projector (projecting means) 2 can projects the given information (light spot) according to the musical score information. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an operation support system and an operation support method for supporting operation by displaying guide information on an object to be operated by a user, particularly when the user operates / learns a complicated machine or musical instrument. The present invention relates to an operation support system and an operation support method for supporting operation and learning by displaying guide information each time.

  2. Description of the Related Art Conventionally, in a museum or an art museum, a customer guidance system for projecting a destination guidance arrow or the like on a floor and instructing an optimal viewing route to a customer is used. In addition, when a customer has a wireless IC card that records destination information, the computer reads the information of the wireless IC card, and the projector projects directions to the customer's feet using arrows or the like to guide directions. Guidance systems that perform this are also in practical use. For example, in a large hospital, when an outpatient walks around the hospital with a wireless IC card indicating internal medicine destination guidance, the route to the internal medicine is projected on the feet of the outpatient, so that the patient is not lost. You can get to the internal medicine window. Furthermore, when a handy projector is illuminated against a target, the camera recognizes the location, and the operation that the projector should perform at that location (for example, the operation that indicates which electronic device is to be plugged into the outlet of the illuminated location) A technique for displaying is also reported.

In recent years, home appliances have become multifunctional, so there are a large number of buttons to be operated on the home appliance main body and remote control. Therefore, home appliances that support the user's operation by causing the button itself to emit light and blink and causing the button to be operated next to emit light or blink are also commercially available. Furthermore, techniques such as an electronic piano and an electronic organ that are configured so that the keyboard to be operated next in accordance with the performance contents have been reported (see, for example, Patent Document 1). There has also been reported an electronic musical instrument technique that not only shows the keyboard to be played next, but also can provide information on the length of notes and the strength of sounds (see, for example, Patent Document 2).
JP-A-8-95560 Japanese Patent Laid-Open No. 2001-184065

  However, the guidance system that guides the road by projecting an arrow on the floor is a simple system that only performs a predetermined road guidance according to a preset pattern, so it can operate and learn complicated machines and instruments. It cannot be developed to technology that supports Naturally, even a system that provides route guidance using a wireless IC card cannot be applied to a technology that supports a complicated operation desired by a customer. In other words, in a system that provides route guidance using a wireless IC card, when there is no means to express the user in an easy-to-understand manner when the user wants to operate a button or the like, only limited information such as projection of an arrow is presented. I can't. In addition, even in home appliances that support operations by emitting and blinking buttons, the buttons cannot be emitted and blinked unless a mechanism for emitting and blinking is incorporated in the buttons themselves. Moreover, in such a method of light emission / flashing of the button, only information to the extent that the light color is changed or the time interval of lighting / flashing is changed can be presented to the user. In addition, for the electronic musical instruments presented in Patent Document 1 and Patent Document 2, a plurality of patterns to be displayed in advance are prepared, and information is displayed by mechanically switching the patterns. Only very limited information can be presented. Similarly, in the technology for displaying the operation content to be performed at the place in the light of the projector, the user can only present simple information to the user so as to indicate the type of electronic device to be plugged into the outlet.

  The present invention has been made in view of the above problems, and when a user operates or learns a complicated device or musical instrument, the guide information that supports the operation is projected from the projector. It is an object of the present invention to provide an operation support system and an operation support method that can be used.

  The present invention has been developed to achieve the above-described object, and is an operation support system for supporting an operation procedure performed by a user using an operation means, and images a nearby region including the operation means. Imaging means for detecting the position of the operating means to be operated by the user from the image information captured by the imaging means, and projecting means for projecting the given information to the detected position of the operating means. Features.

  According to the operation support system of the present invention, when the imaging unit pre-captures a nearby region including the operation unit operated by the user, the projection unit operates the operation unit to be operated by the user from the image information captured by the imaging unit. Since the position of (for example, a piano keyboard) is detected and given information is projected onto the position, the user can operate the operating means (piano keyboard) according to the position where the given information is projected. it can. Accordingly, information indicating an accurate operation procedure can be provided to the user to assist the operation.

  In the present invention, the projecting means sequentially projects given information to the position of the operating means operated by the user according to the operating procedure. That is, according to the operation support system of the present invention, when the user sequentially operates (plays) the operation means (for example, piano keyboard) according to a predetermined procedure, the operation procedure (for example, the keyboard) of the operation means (piano keyboard). The given information is projected in accordance with the timing and order of pressing (hereinafter referred to as “performance order”), so that the user performs a desired operation (for example, piano performance) in accordance with the given information. Can do. Thereby, it is possible to appropriately support the user such as piano practice.

  In the present invention, the projecting means projects given information onto the position of the operating means according to the operation result of the user. For example, if the operation procedure (performance order) is correct, the given information is projected on the operation means (piano keyboard) one after another, and the operation result (for example, how to press the keyboard) is incorrect or ambiguous. In such a case, since the information indicating the content is projected, detailed operation support (performance support) can be provided to the user.

  In the present invention, when the operation result of the user is different from the operation content expected in advance, the projecting means projects the given information for clarifying that. That is, when the operation procedure (performance order) is wrong, given information that points out the error is projected, so that detailed operation support (performance support) can be provided to the user.

  In the present invention, the projection unit projects at least one of character information and image information indicating an operation procedure by the operation unit at or near the position of the operation unit. That is, according to the operation support system of the present invention, not only the given information is projected on the operation means, but also character information and image information (for example, a musical score, etc.) are displayed together, so that more detailed operation support ( Performance support).

  In the present invention, the display contents of the character information and the image information change with the progress of the operation procedure. For example, the display content of the score on the score is changed as the performance progresses, so that the user can practice performance according to the displayed score.

  In the present invention, the character information and the image information are a score or a document and an image including the score. That is, according to the present invention, when a user performs a practice such as a piano, a musical score, a document, an image, and the like are displayed together with given information projected on an operation means (piano keyboard). You can practice performances that are easy to understand.

  In the present invention, the given information is guide information based on illuminated light. That is, since the guide information by the illuminated light is displayed at the operation position of the operation means (piano keyboard etc.), the operation support (performance support) convenient for the user can be performed.

  The operation support method of the present invention is an operation support method for supporting an operation procedure performed by a user using an operation unit, the imaging unit imaging a nearby region including the operation unit, and the imaging unit. A procedure for performing image processing on the image information captured by the image processing to determine a projection position to be projected, and a procedure for acquiring operation procedure information that defines timing and position information for projecting guide information to the projection position; And a procedure for projecting the guide information onto the operation means based on the projection position information and the operation procedure information. With such an operation support method, detailed operation support information can be provided to the user.

  In addition, the operation support method of the present invention further includes a procedure of detecting an operation content of a user, performing image processing, and detecting / analyzing an event generated by the operation result. Thus, if the operation result is detected / analyzed and fed back to the user, the user can perform the next operation more appropriately.

  In the operation support method of the present invention, the guide information is at least one of information indicating a position to be operated on the operation means, character information for supporting the operation, and image information. Thus, if character information and image information are projected as guide information together with position information of the operating means, more appropriate guide information can be provided to the user.

  In the operation support method of the present invention, the character information and the image information change automatically as the operation procedure progresses. That is, more appropriate guide information can be provided to the user by appropriately changing the contents of the character information and image information to be projected as the operation proceeds.

  In the operation support system and the operation support method of the present invention, in order to determine the projection position, the projection target is determined by imaging a projection target with a camera (imaging means) in advance and processing the captured content. Next, information on the operation procedure that determines at which timing the guide information is projected at which timing is acquired. For example, musical score information is obtained for musical instrument performance. Then, the operation content itself (for example, deduction content in terms of a musical instrument) operated by the user according to the projected guide information is detected as an image. Furthermore, the movement of the user's hand or finger is acquired by performing image processing on the detection result of the operation content. Alternatively, an event that occurs as a result of the user's operation (for example, in terms of musical instruments, the pitch or timing of the sound generated as a result of the operation) is detected and analyzed. Thereby, it is possible to perform appropriate operation support (performance support) for the user. In addition to directly illuminating the operation means to be operated, it is also possible to project something like a score. In this case, a user-friendly operation support system can be constructed by changing the display content so that the page is automatically turned as the performance progresses.

  Further, according to the present invention, even if the object is an apparatus that does not have an electrical or optical light emitting means, the operation method and the like can be projected and instructed by the projector. Furthermore, visual feedback information can be displayed by detecting the user's operation content. Further, since the projector is used, various other information can be presented together. Thereby, it is possible to construct an operation support system that is extremely convenient for the user.

  Hereinafter, embodiments of an operation support system according to the present invention will be described in detail with reference to the drawings. First, an outline of an operation support system according to the present invention will be described. The operation support system of the present invention uses a projector to support the operation and performance learning when operating a complex machine or learning the performance of an instrument such as a piano. It is configured to convey guide information to the user by illuminating parts, buttons, keys, keyboards, etc., and projecting additional information such as letters and pictures at the position of the operation target Yes. That is, in the operation support system of the present invention, the projector is not used as a mere projection device on the screen, but is used for providing guide information by projecting necessary information to guide the user.

  In other words, from the viewpoint of projecting guide information desired by a user using a projector, the following several pieces of guide information can be provided to the user by taking an example of musical instrument performance as an example. (1) Illuminate the keyboard to be hit, which is the operation target, with a projector. (2) The display content is changed by changing the display color or blinking depending on whether or not the operation result of the user is correct. (3) The musical instrument and the like are projected together with an operation instruction such as a musical score and a betting keyboard to be operated, thereby assisting the performance of the instrument. The operation support system according to the present invention is not limited to support for playing musical instruments. However, in the following embodiment, in order to facilitate understanding, an example of performing musical instrument assistance using a piano is taken as an example. Give an explanation. That is, several embodiments will be described in the case where performance assistance is performed by brightly illuminating a keyboard to be struck from piano score information as the operation support system of the present invention.

<First Embodiment>
In the first embodiment, a method for determining a position to be projected on a piano and a method for performing a keyboard operation based on the projected information will be described as an operation support system. In the following, in the drawings used in the embodiments, the same components are denoted by the same reference numerals.

  FIG. 1 is a conceptual diagram showing the configuration of piano performance support applied to the operation support system of the first embodiment of the present invention. This operation support system is installed at a position where the user can operate the piano 1, the projector 2 installed at a position where the vicinity of the keyboard of the piano 1 can be illuminated, and the projection range of the projector 2. A process in which the projector 2 illuminates a desired position of the piano 1 at an accurate timing based on the score information recorded on the CD-ROM 4 or the like while controlling the camera 3 and the projector 2 and the camera 3 And a personal computer (hereinafter referred to as a PC) 5 for executing the above.

  Next, a keyboard support operation using the operation support system shown in FIG. 1 will be briefly described. FIG. 2 is a diagram showing a shooting range of the camera in the operation support system shown in FIG. In other words, in the operation support system of FIG. 1, as shown in FIG. 2, the camera 3 is installed at a position where the keyboard 1a, which is the user's operation target, is placed in the photographing field of view. As a result, the camera 3 can always monitor which one of the keys 1a is pressed while imaging.

  FIG. 3 is a diagram showing a projection range of the projector in the operation support system shown in FIG. In FIG. 3, the projected portion of the piano 1 is shaded for easy understanding. Since the projector 2 projects the piano 1 from above, the floor that is a part far from the projector 2 has a wide projection range, and the vicinity of the keyboard 1a that is a close part is narrow.

  FIG. 4 is a conceptual diagram showing a state in which round guide spots are projected from the projector onto five keys in the operation support system shown in FIG. The user can perform performance learning of a desired song by tapping the keyboard 1a according to the guide spots a, b, c, d, e projected from the projector 2 onto the keyboard 1a. For example, since the guide spot lights up (or blinks) in the order of a → e → c → b → d in accordance with a certain song, it is possible to assist the user in learning the performance of the desired song.

  In order for the operation support system shown in FIG. 1 to perform such a performance learning support operation, it is necessary to first calibrate the projection position of the projector 1. Therefore, a method for calibrating the projection position of the projector 1 will be described below.

  FIG. 5 is a block diagram showing a configuration for calibrating the projection position of the projector in the operation support system of the present invention. That is, the user operation support system for calibrating the projection position of the projector includes a projector (projection means) 2, a camera (imaging means) 3, and a PC 5, and the PC 5 is a control unit that controls the entire PC. 11. An image processing unit 12 that performs processing of an image captured by the imaging unit (camera) 3 to determine an operation target position; an operation unit projection that stores operation unit projection position information that is a screen coordinate of the projector (projection unit) 2 A position information DB 13, an operation means shape information DB 14 for storing shape information of operation means such as a keyboard 1 a, and a calibration projection pattern information DB 15 for storing a calibration projection pattern taken in by the imaging means (camera) 3 are provided. DB is an abbreviation for database.

  Next, an operation when the projection position of the projector (projection means) 2 is calibrated will be described. 5, first, an image of the operation target (keyboard 1a) is picked up by the image pickup means (camera) 3 and transmitted to the PC 5 as shown in FIG. Then, the PC 5 transfers the imaging information to the image processing unit 12 via the control unit 11. Thus, the image processing unit 12 extracts feature points from the image captured by the imaging unit (camera) 3 based on the shape information of the operation unit (that is, the keyboard 1a) stored in the operation unit shape information DB 14. Then, matching is performed to determine the position of the operation target in the coordinates of the imaging means (camera) 3 (that is, the position of the keyboard 1a), and the position coordinates are mapped to its own memory. At this time, the control unit 11 examines which pixel of the projection coordinates in the projector (projection means) 2 corresponds to the position coordinates of the operation means (keyboard 1a) imaged by the imaging means (camera) 3 to obtain the imaging means ( The correspondence between the position coordinates of the operation means (keyboard 1a) by the camera 3 and the projection coordinates of the projector (projection means) 2 is recorded in the operation means projection position information DB 13.

  FIG. 6 is a conceptual diagram illustrating a state in which the imaging unit (camera) captures the projection result and performs calibration in the configuration diagram for calibrating the projection position illustrated in FIG. 5. That is, as illustrated in FIG. 6, the control unit 11 projects the calibration projection pattern of the operation unit (keyboard 1 a) onto the projection unit (projector) 2 and causes the imaging unit (camera) 3 to capture the projection result. More specifically, as shown in FIG. 6 (b), the entire image of the keyboard 1a in the piano 1, which is the projection range of the projection means (projector) 2, is picked up by the image pickup means (camera) 3, and FIG. As shown in FIG. 5, the highlight portion S is captured by the imaging means (camera) 3 and used as the projection pattern information for calibration.

  That is, FIG. 6A is a calibration projection pattern for calibrating the projection position, and only a desired portion (that is, the highlight portion S) of the projection range by the projection means (projector) 2 is brightened. By imaging it with the imaging means (camera) 3, the bright part of the screen coordinates of the projection means (projector) 2 corresponds to which part of the camera coordinates, and as a result, which part of the operating means (keyboard 1a) Can be determined. In this manner, the light color portion of the projection pattern is sequentially scanned on the keyboard 1a of FIG. 6B, thereby making it possible to coordinate the projection means (projector) 2 and the operation means (keyboard 1a) to be projected. The correspondence between the positions can be checked. For example, the keyboard 1a in FIG. 6B is scanned by dividing it into 16 parts, and the correspondence between the coordinates of the projection means (projector) 2 and the position of the operation means (keyboard 1a) to be projected is calibrated. In this way, the control unit 11 shown in FIG. 5 stores the operation unit projection position information in the operation unit projection position information DB 13 at the time of calibration.

  FIG. 7 is a block diagram showing a configuration for performing a keyboard support operation using the operation support system in the first embodiment of the present invention. That is, the operation support system according to the first embodiment for performing the support operation of the keyboard includes a projector (projection means) 2, a camera (imaging means) 3, and a PC 5, and the PC 5 is the entire PC. Control unit 11 that performs control, projection image generation unit 16 that generates a projection image by adding information such as color, size, and shape to an image to be projected, and operations that are screen coordinates of the projector (projection means) 2 An operation means projection position information DB 13 for storing means projection position information and a display projection pattern information DB 17 for storing information such as the color, size and shape of the image to be projected are provided.

  Next, an operation for performing keyboard operation support using the user operation support system of FIG. 7 will be described. When actually performing the support operation of the keyboard, the user firstly stores the musical score information to be played from a medium such as a CD-ROM or a network in a data format of, for example, MIDI (Musical Instrument Digital Interface) 18 format or from the network. Input to the control unit 11. The MIDI data format is data in a format that describes whether an event such as key-on (a command to start sound by pressing the keyboard down) or key-off (release the keyboard to stop the sound) occurs at the time of the keyboard. This data is suitable as operation support information. The control unit 11 causes the projection image generation unit 16 to generate an image to be projected based on the input musical score information and the operation unit projection position information stored in the operation unit projection position information DB 13 at the time of calibration. Then, an image to be projected (that is, a guide spot on the keyboard 1a) is projected from the projection unit (projector) 2 to the operation unit (keyboard 1a) via the control unit 11.

  Note that the projection image generation unit 16 acquires data such as color, size, and shape from the display projection pattern information DB 17 and adds the information such as color, size, and shape to the image to be projected and causes the image to be projected. You can also. Further, the imaging means (camera) 3 at the time of operation always monitors whether or not the user has performed an operation as supported. In this way, in the keyboard support operation using the operation support system, the image to be projected is generated based on the musical score information, the operation means projection position information generated at the time of calibration, the display projection pattern information, and the like. By projecting onto the operation means (keyboard 1a) by the projector (projection means) 2, the operation of the keyboard 1a can be appropriately supported.

<Second Embodiment>
In the second embodiment, an example will be described in which a keyboard operation performed by the user is detected and a display corresponding to the operation is performed. As a means for detecting a keyboard operation, image determination may be performed using an imaging means (camera), or a sound as a result of the keyboard operation may be detected to analyze a frequency or the like.

  FIG. 8 is a conceptual diagram showing the configuration of a piano performance applied to the operation support system according to the second embodiment of the present invention. The operation support system of the second embodiment shown in FIG. 8 is different from the first embodiment shown in FIG. 1 only in that a microphone 6 is added. Therefore, the description of the overlapping part is omitted. In FIG. 8, it is detected which keyboard is operated by the user during performance based on at least one of the image information captured by the camera 3 and the sound information recorded by the microphone 6.

  FIG. 9 is a block diagram showing a configuration for performing a keyboard support operation using the operation support system in the second embodiment of the present invention. That is, the operation support system according to the second embodiment for performing the keyboard support operation includes the projector (projection unit) 2, the user operation detection unit 19 that is a sensor such as the camera 3 and the microphone 6, and the PC 5. In the configuration, the PC 5 includes a control unit 11 that controls the entire PC, a projection image generation unit 16 that generates a projection image by adding information such as color, size, and shape to the image to be projected, and a projector (projection). Means) operation means projection position information DB 13 for storing operation means projection position information which is the screen coordinates of 2; display projection pattern information DB 17 for storing information such as the color, size and shape of the image to be projected; User operation determination means 20 for determining which keyboard is operated.

  Note that if the image information is detected, the user operation determination means 20 takes the keyboard operation by taking the difference between the previously stored reference image that does not show the user's hand and the detected image information. It is possible to detect where the hand is inside. Further, it is possible to calculate which keyboard is operated by extracting the position of the edge of the keyboard from the image information.

  On the other hand, if sound information is detected by the microphone 6, frequency analysis of the input sound information is performed, and a frequency component when correctly operated is estimated from the score information used in the operation, and the sound information By taking the difference between the frequency analysis and the frequency component obtained from the score information, it is possible to determine whether the user's keyboard operation is correct or incorrect. Note that it is also possible to acquire a user operation result with a microphone, a camera, or the like, determine whether the operation is correct, and change the image content to be projected according to the determination result.

  In addition, when a user's keyboard operation is detected by a combination of image information detection and sound information detection and an error is found as a result (that is, information estimated from musical score information and operation information based on hand movement and sound) (If they are different), you can change the color of the wrongly projected keyboard, or change the color of the notes projected as the score.

  FIG. 10 is a diagram showing a projection screen of the projector showing a state in which the musical score information is projected together with the piano keyboard in the operation support system of the present invention. As shown in FIG. 10, not only the spotlight projection of the keyboard but also the image of the score is projected together, so that the keyboard operation can be supported more finely.

  According to the operation support system of the present invention, it can be widely used for educational purposes for newcomers such as assisting in playing and practicing musical instruments, assisting in the operation of complex mechanical devices, and practicing keyboard operations on personal computers.

It is a conceptual diagram which shows the structure of the piano performance assistance applied to the operation assistance system of 1st Embodiment in this invention. It is a figure which shows the imaging | photography range of the camera in the operation assistance system shown in FIG. It is a figure which shows the projection range of the projector in the operation assistance system shown in FIG. In the operation support system shown in FIG. 1, it is a conceptual diagram which shows the state which projected the round guide spot on five keyboards from the projector. It is a block diagram which shows the structure for calibrating the projection position of a projector in the operation assistance system of this invention. In the block diagram for calibrating the projection position shown in FIG. 5, it is a conceptual diagram which shows the state which an imaging means (camera) takes in a projection result and calibrates. FIG. 3 is a block diagram showing a configuration for performing a keyboard support operation using the operation support system in the first embodiment of the present invention. It is a conceptual diagram which shows the structure of the piano performance applied to the operation assistance system of 2nd Embodiment in this invention. In the 2nd Embodiment of this invention, it is a block diagram which shows the structure for performing the assistance operation | movement of a keyboard using an operation assistance system. In the operation support system of this invention, it is a figure which shows the projection screen of the projector which shows the state which also projects musical score information with a piano keyboard.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Piano, 1a Keyboard, 2 Projector (projection means), 3 Cameras (imaging means), 4 CD-ROM, 5 PC, 11 Control part, 12 Image processing part, 13 Operation means Projection position information DB, 14 Operation means shape information DB, 15 Calibration projection pattern information DB, 16 Projected image generation unit, 17 Display projection pattern information DB, 18 Recording medium (CD-ROM) on which MIDI data is recorded, 19 User operation detection means, 20 User operation determination means

Claims (12)

An operation support system for supporting an operation procedure performed by a user using an operation means,
Imaging means for imaging a nearby region including the operation means;
A projection unit that detects a position of the operation unit to be operated by a user from image information captured by the imaging unit, and projects given information on the detected position of the operation unit;
An operation support system comprising:   The operation support system according to claim 1, wherein the projection unit sequentially projects the given information to a position of the operation unit operated by a user according to an operation procedure.   The operation support system according to claim 1, wherein the projection unit projects given information onto a position of the operation unit according to a user operation result.   4. The operation support system according to claim 3, wherein when the operation result of the user is different from the operation content expected in advance, the projecting unit projects given information for clarifying the fact.   5. The projection unit according to claim 1, wherein the projection unit projects at least one of character information and image information indicating an operation procedure by the operation unit to a position of the operation unit or the vicinity thereof. The operation support system described in Crab.   The operation support system according to claim 5, wherein display contents of the character information and the image information change as an operation procedure progresses.   The operation support system according to claim 5 or 6, wherein the character information and the image information are a score or a document and an image including the score.   The operation support system according to any one of claims 1 to 7, wherein the given information is guide information based on illuminated light. An operation support method for supporting an operation procedure performed by a user using an operation means,
A procedure in which an imaging unit images a region near the operation unit;
A procedure for performing image processing on image information captured by the imaging unit and determining a projection position to be projected;
A procedure for acquiring operation procedure information that defines timing and position information for projecting guide information to the projection position;
An operation support method comprising: a step of projecting the guide information onto the operation means based on the projection position information and the operation procedure information.   The operation support method according to claim 9, further comprising a procedure of detecting an operation content of a user to perform image processing, and detecting and analyzing an event that occurs according to an operation result.   The operation support method according to claim 10, wherein the guide information is at least one of information indicating a position to be operated in the operation unit, character information for supporting the operation, and image information.   The operation support method according to claim 11, wherein the character information and the image information change automatically as an operation procedure progresses.

Images