JP5401386B2 - Information processing system and information processing method - Google Patents

Description

  The present invention relates to a technique capable of superimposing and displaying an image in real space and computer graphics image data.

  Several techniques are known to support piano practice. For example, Patent Document 1 discloses an electronic keyboard instrument that has a function of illuminating a key of a corresponding sound together with a melody of a built-in song. Therefore, it is possible to express the melody of a song simply by pressing a shining key using this function.

  In addition, Non-Patent Document 1 discloses a system that supports the acquisition of fingering for piano performance utilizing fingering recognition technology. This system takes image data taken by a camera and MIDI (Musical Instrument Digital Interface) data as input, performs fingering recognition and system operation analysis based on these data, and projects a musical score at a fixed position above the keyboard. This is a technology that projects the keystroke position and fingering on the keyboard.

Japanese Patent No. 3465377

Japanese software science world, “Design and implementation of piano performance learning support system using fingering recognition technology”, Internet <http://www.wiss.org/WISS2009Proceedings/papers/paper0008.pdf>

  However, these conventional techniques are very complicated because the viewpoint is repeatedly transferred between the hand movement and the musical score. For example, in the case of a piano, there are 88 keys on the keyboard. If you think that the melody of the bass key has continued, only the treble key may continue from the next measure. In this case, the line of sight moves greatly from the bass key area to the treble key area in accordance with the hand movement. In addition, since the musical score is viewed in the meantime, a more complicated point of view occurs.

  Therefore, an object of the present invention is to make it possible to learn operation of equipment without stress.

An information processing system according to the present invention includes an augmented reality presentation device capable of combining and displaying computer graphics image data on an image in a real space, and time-series information related to operation of a device existing in the real space. The augmented reality presentation device includes: a first image data generation unit configured to generate image data of a body part on which a user operates the device; and First transmission means for transmitting image data of the body part generated by the image data generation means to the management apparatus, wherein the management apparatus is time-sequential information relating to the operation of the device. Specifying means for specifying a current operation time point in the user's operation, and acquisition means for acquiring time-series information relating to the operation of the device corresponding to the current operation time point , Based on the image data of the body part, and identifying means for identifying the position of the body part, in the vicinity of the position of the body part identified by said identification means, to said current operation acquired by the acquisition unit First augmented reality presentation device for instructing the augmented reality presentation device to present time-series information related to the operation of the device corresponding to the time, and the augmented reality presentation device In response to an instruction from one instruction means, computer graphics image data indicating time-series information related to the operation of the device corresponding to the current operation time point is displayed as an image in the vicinity of the position of the body part in the real space. It further comprises first presenting means for composing and presenting.
An information processing method according to the present invention includes an augmented reality presentation device capable of combining and displaying computer graphics image data on a real space image, and time-series information relating to operation of a device existing in the real space. An information processing method executed by a management device that manages the image, wherein the augmented reality presentation device includes an image data generation step of generating image data of a body part on which a user operates the device, and the image data generation Transmitting the image data of the body part generated by the step to the management device, wherein the management device includes the time series information on the operation of the device in the operation of the user. A specifying step for specifying the current operation time point, and an acquisition step for acquiring time-series information related to the operation of the device corresponding to the current operation time point. And flop, based on the image data of the body part, wherein the identification step of identifying the position of the body part, the identification in the vicinity of the location of the identified said body part by step, the obtained by the obtaining step current An instruction step for instructing the augmented reality presentation device to present time-series information related to the operation of the device corresponding to the operation point of the augmented reality presentation device, the augmented reality presentation device comprising: In response to the instruction, computer graphics image data indicating time-series information related to the operation of the device corresponding to the current operation time point is synthesized and presented in an image near the position of the body part in the real space. And a presentation step.

  According to the present invention, it is possible to learn the operation of a device without stress.

It is a figure showing roughly the composition of the learning support system concerning the embodiment of the present invention. It is a figure which shows the hardware constitutions of a learning assistance management server. It is a figure which shows the hardware constitutions of HMD and a small camera. It is a flowchart which shows the flow of a process of the learning assistance system which concerns on embodiment of this invention. It is a flowchart which shows the flow of the musical score AR display process of HMD. It is a flowchart which shows the flow of fingering AR display processing of HMD. It is a figure which shows the example of AR display of the musical score part corresponding to the present performance position. It is a figure which shows the AR display example of fingering corresponding to a present performance position.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram schematically showing a configuration of a learning support system according to an embodiment of the present invention. The learning support system according to the present embodiment shown in FIG. 1 includes a learning support management server 6 that manages musical score data, fingering data, and the like, which will be described later, to support the user's learning of piano performance. The learning support management server 6 is communicably connected to the small camera 2 and HMD (Head Mounted Display) 1 and the piano 5 via a wireless communication line. The HMD 1 is a spectacle type and is worn on the user's head. The HMD 1 produces an expanded sense of reality by displaying computer graphics image data at a position that matches optical images that are optically transmitted and projected on the lens portion of the HMD 1. In the following description, AR (Augmented Reality) is a technique for superimposing and displaying computer graphics image data managed by a computer such as characters, graphics, still images, and moving images on a real-space video and providing supplementary information. Reality) display. The small camera 2 is attached to the HMD 1 so that image data can be taken at an angle of view close to the user's field of view. The captured image data is transmitted to the learning support management server 6 via a wireless communication line. The HMD 1 includes various methods such as a non-transmissive type, a video transmissive type, and an optical transmissive type. The composite display (AR display) is to superimpose and display computer graphics image data at a matching position on the real space image displayed on the lens portion of the HMD 1 by any method. In the case of the type method, the image is superimposed and displayed on the image of the real space seen through, and in the case of the video transmission type, the image is displayed on the image of the real space captured by the video camera.

  In the present embodiment, the eyeglass-type HMD 1 is cited as an example of an AR display device that is a device that performs AR display, but is not limited thereto. For example, a mobile terminal AR display device that displays a real-space image captured by a camera on a liquid crystal display or the like, and displays and displays computer graphics image data on the displayed real-space image may be used. It may be a head-up display AR display device that is installed in the line-of-sight direction of the driver's seat of the vehicle and displays computer graphics image data in a composite image of real space that is optically transmitted and projected. In addition, when computer graphics image data is displayed in alignment with an image in the real space, coordinate alignment with the computer graphics image data in the image in the real space is performed. In the present embodiment, a marker is attached to an object in the real space, the marker is photographed with a camera, the position of the marker is detected from the photographed image data, and computer graphics are used so as to match the detected position. Synthesize image data. Further, as another coordinate matching method, it is estimated where the user wearing the HMD is looking in the real space based on the position (latitude, longitude) and posture (direction, elevation angle) of the HMD, and the estimated position The computer graphics image data may be synthesized so as to match the above. Furthermore, the position of the object in the real space may be detected by analyzing the image data in the real space captured by the imaging unit, and the computer graphics image data may be synthesized so as to match the detected position.

  A marker 4 is attached to each key of the piano 5 played by the user. By capturing the marker 4 with the small camera 2 and analyzing the captured image data, the key position can be specified. It should be noted that standard information such as the keyboard size of the piano 5 is provided in the learning support management server 6 in advance, a marker is attached only to the reference key, the position of the key is specified, and this piano is used as a base point. The position of each key may be specified from the standard information. The marker 3 is also attached to the user's hand, and the position of the user's hand can be specified by photographing the marker 3 with the small camera 2 and analyzing the photographed image data.

  The learning support management server 6 includes a musical score data acquisition / transmission unit 61, a current performance position determination unit 62, a fingering data acquisition / transmission unit 63, a musical score data storage unit 64, a MIDI data storage unit 65, a fingering data storage unit 66, and A sound-marker correspondence management file 67 is provided. The musical score data storage unit 64 stores musical score data of a plurality of music pieces. The musical score data acquisition / transmission unit 61 includes musical score data corresponding to the current performance position of the user (hereinafter referred to as the current performance position) among the musical score data of the music selected as the performance target by the user. Get from. The musical score data acquisition / transmission unit 61 acquires image data of the user's hand photographed by the small camera 2 and performs image analysis of the marker 3 attached to the user's hand, thereby allowing the user's hand. The position of is detected. Then, the musical score data acquisition / transmission unit 61 transmits the detected musical location information and musical score data corresponding to the current performance position to the HMD 1 and also the computer graphics image of the musical score portion corresponding to the current performance position. The HMD 1 is instructed to synthesize the data near the user's hand and display the AR. The HMD 1 synthesizes the computer graphics image data representing the musical score portion corresponding to the current performance position with the video near the user's hand in the real space and displays the AR.

  The MIDI data storage unit 65 stores MIDI (Musical Instrument Digital Interface) data of a plurality of music pieces. Here, the MIDI data is music performance data and is expressed based on the MIDI standard. The MIDI standard is a data standard for controlling an electronic musical instrument from the outside so that the musical instrument is played. The MIDI standard is defined so that data about each sound constituting a musical piece is input to the electronic musical instrument or the like in time series. MIDI data is the score information such as the pitch, size, length of the sound, the interval from the start time of the sound to the start time of the next sound, etc. Contains. The MIDI data storage unit 65 stores MIDI data generated based on an accurate piano performance, and the user's current performance position and the like by comparison with MIDI data generated according to the user's piano performance described later. Is determined. In the following description, the MIDI data stored in the MIDI data storage unit 65 is referred to as reference MIDI data. When the user plays the piano 5, MIDI data indicating the performance content is generated in the piano 5 and transmitted to the learning support management server 6. In the following description, the MIDI data generated by the piano 5 in accordance with the performance of the user is referred to as performance MIDI data. The current performance position determination unit 62 acquires the reference MIDI data of the music selected by the user from the MIDI data storage unit 65, and compares the performance MIDI data with the reference MIDI data to determine the current performance position. The determination result of the current performance position is output to the musical score data acquisition / transmission unit 61 and a fingering data acquisition / transmission unit 63 described later.

  The fingering data storage unit 66 stores fingering data of a plurality of music pieces. Fingering data is data indicating fingering when playing a musical piece on a piano. The fingering data acquisition / transmission unit 63 acquires fingering data corresponding to the current performance position from the fingering data storage unit 66 among the fingering data of the music selected as a performance target by the user, and further performs the fingering data Computer graphics image data corresponding to the data is acquired. Further, the fingering data acquisition / transmission unit 63 acquires the image data of the keyboard photographed by the small camera 2 and analyzes the image of the marker 4 attached to the keyboard, thereby obtaining the sound at the current performance position. Detect the position of the corresponding key. Although details will be described later, the fingering data acquisition / transmission unit 63 refers to the sound-marker correspondence management file 67 in which the sound is associated with the marker 4 on the keyboard, and the marker 4 corresponding to the sound at the current performance position. Determine. Then, the fingering data acquisition / transmission unit 63 synthesizes the fingering computer graphics image data corresponding to the current performance position with the key image in the real space to which the marker 4 is attached and displays the AR. HMD1 is instructed. The HMD 1 synthesizes the fingered computer graphics image data corresponding to the current performance position with the video of the corresponding key in the real space and displays the AR.

  FIG. 2A is a diagram illustrating a hardware configuration of the learning support management server 6. The CPU 201 comprehensively controls each device and controller connected to the system bus. A ROM 203 or an HD (hard disk) 207 stores a basic input / output system (BIOS) that is a control program of the CPU 201, an operating system program, a program for the processing shown in FIG. Yes.

  In the example of FIG. 2A, the HD 207 is arranged inside the learning support management server 6. However, as another embodiment, a configuration corresponding to the HD 207 is arranged outside the learning support management server 6. It is good also as a structure. Also, the program for performing the processing shown in FIG. 3 according to the present embodiment is recorded on a computer-readable recording medium such as a flexible disk (FD) or a CD-ROM, and is supplied from the recording medium. Alternatively, it may be configured to be supplied via a communication medium such as the Internet.

  The RAM 202 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program necessary for execution of processing into the RAM 202 and executing the program.

  The HD 207 and the FD 206 function as an external memory. The CPU 201 implements various operations by loading a program necessary for execution of processing into the RAM 202 and executing the program.

  The disk controller 208 controls access to external memories such as the HD 207 and the FD 206. The communication I / F controller 204 is connected to the Internet or a LAN, and controls communication with the outside by, for example, TCP / IP.

  The display controller 208 controls image display on the display 209.

  The KB controller 210 receives an operation input from the KB (keyboard) 211 and transmits it to the CPU 201. Although not shown, in addition to the KB 211, a pointing device such as a mouse can also be applied to the learning support management server 6 according to the present embodiment as a user operation means.

  The musical score data acquisition / transmission unit 61, the current performance position determination unit 62, and the fingering data acquisition / transmission unit 63 shown in FIG. 1 are stored in, for example, the HD 207 and loaded into the RAM 202 as necessary, and execute the program. This is a configuration realized by the CPU 201. The musical score data storage unit 64, the MIDI data storage unit 65, and the fingering data storage unit 66 have a configuration corresponding to a partial storage area of the HD 207 and the RAM 202. The sound-marker correspondence management file 67 is stored in the HD 207 or the RAM 202.

  FIG. 2B is a diagram illustrating a hardware configuration of the HMD 1 and the small camera 2. The CPU 301 comprehensively controls each device and controller connected to the system bus. The ROM 302 stores, for example, processing programs shown in FIGS. 3 to 5 executed by the HMD 1 and the small camera 2. Note that the program for performing the processes shown in FIGS. 3 to 5 may be supplied via a communication medium such as the Internet.

  The RAM 304 functions as a main memory, work memory, and the like for the CPU 301. The CPU 301 implements various operations by loading a program necessary for execution of processing into the RAM 304 and executing the program. The communication I / F controller 303 is connected to the Internet or a LAN, and controls communication with the outside by TCP / IP, for example.

  The imaging unit 305 converts a subject image incident through an optical lens and formed on the imaging element into an electrical signal, and outputs moving image data or still image data. The display controller 306 controls image display on the display 307. In this embodiment, an optically transmissive head-mounted display is employed, the display 307 is configured by a half mirror, and the user who wears the camera can view the outside through the display 307. The imaging unit 305 has a configuration corresponding to the small camera 2 in FIG.

FIG. 3 is a flowchart showing a processing flow of the learning support system according to the present embodiment.
First, in step S <b> 101, the user photographs his / her hand with the small camera 2. In step S <b> 102, the image data at hand taken by the small camera 2 is transmitted to the learning support management server 2.

  In step S <b> 201, the learning support management server 6 determines whether image data on hand of the user has been received. This determination is made based on whether or not the marker 3 is included in the received image data. If the marker 3 is included in the received image data and it is determined that the image data at hand is received, the process proceeds to step S202. On the other hand, when the image data itself is not received or the marker 3 is not included in the received image data, the learning support management server 6 stands by until receiving the image data at hand.

  After the user captures the image data at hand, the user selects music and starts playing the piano. As the music selection operation, for example, an operation of pressing a music selection button provided on the piano 5 is performed on the piano 5. When a music selection operation is performed on the piano 5, selection information including a user ID and a music ID is transmitted from the piano 5 to the learning support management server 6.

  After the music selection operation, the user performs a piano performance of the music. In step S <b> 103, the piano 5 generates performance MIDI data according to the performance of the user and transmits it to the learning support management server 6. In step S104, the piano 5 determines whether or not the user's performance has ended. When the performance ends, the process ends. On the other hand, if the performance has not ended, the process returns to step S103, and the generation and transmission of performance MIDI data are continued.

  In step S202, the learning support management server 6 determines whether or not performance MIDI data has been received from the piano 5. If performance MIDI data is received, the process proceeds to step S203. On the other hand, when the performance MIDI data is not received, the learning support management server 6 stands by until the performance MIDI data is received.

  In step S203, the current performance position determination unit 62 acquires reference MIDI data corresponding to the music ID included in the selection information from the MIDI data storage unit 65, and compares the reference MIDI data with the performance MIDI data. Determine the current performance position. If the user ID included in the header information of the local image data received in step S201 and the user ID included in the selection information are determined before this processing, and both match, The process of S203 is executed.

  In step S204, the musical score data acquisition / transmission unit 61 acquires the musical score data corresponding to the current performance position from the musical score data storage unit 64 among the musical score data of the music selected by the user.

  In step S205, the musical score data acquisition / transmission unit 61 detects the position of the user's hand by performing image analysis of the marker 3 from the hand image data received in step S201. In step S209, the musical score data acquisition / transmission unit 61 transmits the detected positional information of the user and musical score data corresponding to the current performance position to the HMD 1 and computer graphics of the musical score portion corresponding to the current performance position. The HMD 1 is instructed to synthesize the image data near the user's hand and display the AR (musical score AR display instruction). In addition, the following steps S206 to S208 are executed in parallel with steps S204, S205, and S209.

  In step S206, the fingering data acquisition / transmission unit 63 acquires fingering data corresponding to the current performance position from the fingering data storage unit 66 among the fingering data of the music selected by the user, and further acquires the fingering data. Computer graphics image data corresponding to fingering data is acquired. In step S207, the fingering data acquisition / transmission unit 63 acquires the sound at the current performance position in the music from the reference MIDI data, refers to the sound-marker correspondence management file 67, and on the keyboard corresponding to the sound. The marker 4 is determined. Note that the sound at the current performance position may be acquired by frequency analysis.

  Here, since the user is playing while looking at his / her hand, the small camera 2 photographs the marker 4 on the keyboard together with the user's hand. In step S208, the fingering data acquisition / transmission unit 63 determines the position of the marker 4 determined in step S207 from the image data captured by the small camera 2, and corresponds to the position information of the marker 4 and the current performance position. The computer graphics image data of the fingering is transmitted to the HMD 1, and the computer graphics image data of the fingering corresponding to the current performance position is combined with the video of the key to which the marker 4 is attached to display the AR. The HMD 1 is instructed to do this (fingering AR display instruction).

  In step S210, the learning support management server 6 determines whether or not reception of performance MIDI data has been completed. When the reception of the performance MIDI data is completed, the process ends. On the other hand, if reception of the performance MIDI data is not yet completed, the processing from step S203 is executed for the performance MIDI data to be received next.

  FIG. 4 is a flowchart showing the flow of the musical score AR display process of the HMD 1. In step S <b> 105, the HMD 1 determines whether a musical score AR display instruction has been received from the learning support management server 6. When the musical score AR display instruction is received, the process proceeds to step S106. On the other hand, when the music AR display instruction is not received, the HMD 1 stands by until the music AR display instruction is received.

  In step S <b> 106, the HMD 1 uses the position information of the user's hand received from the learning support management server 6 and the musical score data corresponding to the current performance position to obtain computer graphics image data of the musical score portion corresponding to the current performance position. AR display is performed by synthesizing the image near the user's hand in the real space.

  In step S <b> 107, the HMD 1 determines whether or not the reception of the musical score AR display instruction from the learning support management server 6 has been completed. When the reception of the musical score AR display instruction is completed, the process ends. On the other hand, if reception of the musical score AR display instruction has not been completed, the process returns to step S105, and processing corresponding to the AR display instruction for the next musical score part is performed.

  FIG. 5 is a flowchart showing the flow of the fingering AR display process of the HMD 1. In step S <b> 108, the HMD 1 determines whether or not a fingering AR display instruction has been received from the learning support management server 6. If a fingering AR display instruction is accepted, the process proceeds to step S109. On the other hand, when the fingering AR display process is not received, the HMD 1 stands by until a fingering AR display instruction is received.

  In step S109, the HMD 1 uses the position information of the marker 4 received from the learning support management server 6 and the computer graphics image data of the fingering corresponding to the current performance position, and the computer graphic of the fingering corresponding to the current performance position. The image data is synthesized with the video of the key to which the marker 4 is attached and displayed in AR.

  In step S110, the HMD 1 determines whether or not the acceptance of the fingering AR display instruction from the learning support management server 6 has been completed. When the acceptance of the fingering AR display instruction is completed, the process ends. On the other hand, if acceptance of the fingering AR display instruction has not been completed, the process returns to step S109, and a process corresponding to the AR display instruction for the next fingering is performed.

  FIG. 6 is a diagram showing an AR display example of the musical score portion corresponding to the current performance position. As shown in FIG. 6, the HMD 1 AR displays the computer graphics image data 62 of the musical score portion corresponding to the current performance position on the video near the user's hand 61. Thereby, for example, even if the performance key is moved, it is possible to perform while always capturing the musical score in the field of view without changing the line of sight. Therefore, according to this embodiment, it becomes possible to learn the operation of the piano 5 without stress.

  In the present embodiment, the musical score portion corresponding to x measures (x is an arbitrary numerical value set in advance) before and after the current performance position is AR-displayed. In other words, in step S209, musical score data for x bars before and after the current performance position is transmitted from the learning support management server 6 to the HMD1. In step S107, the HMD 1 AR-displays the computer graphics image data of the musical score portion for x bars before and after the current performance position based on the musical score data. In this case, the musical score portion at the current performance position may be highlighted so as to stand out from the other musical score portions. As a result, the current performance position is clarified, and the front-rear relationship of the musical score with respect to the current performance position can be grasped.

  FIG. 7 is a diagram illustrating an AR display example of a fingering corresponding to the current performance position. As shown in FIG. 7, the HMD 1 synthesizes fingering computer graphics image data 710 and 72 corresponding to the current performance position with the video of the key corresponding to the sound at the current performance value position, and displays the AR. In addition, 73 and 74 of FIG. 7 have shown the finger of the user who is performing. The learning support management server 6 includes fingering data in which the fingering order associated with the time information of the MIDI data is set for each piece of music in the fingering data storage unit 66, and the time of the performance MIDI data is provided. The fingering corresponding to the current performance position can be specified from the fingering data based on the information. For example, in the fingering data of the music A, the music A is played as follows: “R1 (thumb of right hand) → R2 (index finger of right hand) → R5 (little finger of right hand) → L3 (middle finger of left hand)”. The order of fingering is set.

  In the learning support management server 6, computer graphics image data is managed for each fingering (R1 (right thumb) -computer graphics image data 1, R2 (right index finger) -computer graphics. Image data 2, ...). When the learning support management server 6 determines the current performance position, the learning support management server 6 identifies the fingering corresponding to the current performance position from the fingering data, acquires the computer graphics image data of the fingering, and transmits it to the HMD 1. As a result, the HMD 1 can AR display the fingering computer graphics image data corresponding to the current performance position. As described above, the computer graphics image data of the fingering is synthesized and displayed at the key position corresponding to the sound at the current performance position. As a result, when the user looks at the hand over the HMD 1, the computer graphics image data of the correct fingering is displayed AR on the hand, and whether or not the player can perform with the correct fingering is intuitively and visually confirmed. I can grasp it.

  In the above example, computer graphics image data is managed for each fingering. However, as another embodiment, computer graphics image data may be managed in association with a combination before and after fingering. That is, R1 in the flow of R2 (right hand index finger) → R1 (right hand index finger) → R2 (right hand index finger) and R3 (right hand index finger) → R1 (right hand index finger) → R2 (right hand index finger) Different computer graphics image data is managed with R1 in the flow, and even when computer graphics image data of the same fingering R1 is specified, different computer graphics image data is specified according to the order of fingering Is done.

  Moreover, a marker for identifying which finger is on each finger of the user is pasted, and the learning support management server 6 manages the marker and fingering in association with each other. The fingering of the user can be specified by analyzing the image of the marker of the photographed user's finger. Then, by comparing the specified fingering with the fingering set in the fingering data, it is possible to determine whether or not there is an error in fingering. You may make it AR display to the vicinity of the finger which there existed.

  Further, the learning support management server 6 can determine whether or not there is an error in the keystroke position by comparing the performance MIDI data and the reference MIDI data. If there is a mistake, the learning support management server 6 instructs the HMD 1 to display the AR in the AR, and the HMD 1 synthesizes the fact with the video near the finger having the mistake in response to the instruction. AR display may be used. As another embodiment, the user's performance sound may be subjected to voice analysis or frequency analysis, and an error in the keystroke position may be detected by comparing the analysis result with reference MIDI data.

  In the above-described embodiment, MIDI data is used to determine the current performance position or to detect a performance error. However, the pitch, volume, and length of the sound, and the next time from the start point of the sound, are used. It is not necessary to limit to MIDI data as long as it includes information such as the interval until the start of the sound. In the above-described embodiment, an example applied to piano performance learning support is described. However, the present invention is not limited to this, and other instruments such as clarinet and flute and PC operations are operated. It can be applied to learning support.

  1: HMD, 2: Small camera, 3, 4: Marker, 5: Piano, 6: Learning support management server, 61: Music score data acquisition / transmission unit, 62: Current performance position determination unit, 63: Fingering data acquisition / Transmission unit, 64: Music score data storage unit, 65: MIDI data storage unit, 66: Fingering data storage unit, 67: Sound-marker correspondence management file

Claims (7)

An augmented reality presentation device capable of combining and displaying computer graphics image data on video in a real space, and a management device for managing time-series information related to operation of devices existing in the real space An information processing system,
The augmented reality presentation device includes:
First image data generating means for generating image data of a body part on which the user operates the device;
First transmission means for transmitting image data of the body part generated by the first image data generation means to the management device;
The management device
Among the time-series information related to the operation of the device, a specifying means for specifying a current operation time point in the user operation;
Acquisition means for acquiring time-series information related to the operation of the device corresponding to the current operation time;
Identification means for identifying the position of the body part based on the image data of the body part;
The augmented reality so as to present, in the vicinity of the position of the body part identified by the identifying unit, time-series information related to the operation of the device corresponding to the current operation time acquired by the acquiring unit . First instruction means for instructing the presentation device;
The augmented reality presentation device includes:
In response to an instruction from the first instruction means, computer graphics image data indicating time-series information related to the operation of the device corresponding to the current operation time point, in the vicinity of the position of the body part in the real space An information processing system further comprising first presentation means for synthesizing and presenting the video. The augmented reality presentation device includes:
Second image data generation means for generating image data of the device;
Second transmission means for transmitting the image data of the device generated by the second image data generation means to the management device;
The management device
Analyzing the image data before Symbol equipment, and before Symbol judging means for judging the operation portion of the device corresponding to the current operation point,
The time-series information when relating to the operation of the device corresponding to the current operation point, the augmented reality so that synthesized and presented on the video of the operation portion of the device corresponding to the time the current operation in the real space Second instruction means for instructing the presentation device;
The augmented reality presentation device includes:
In response to an instruction by the second instruction means, the computer graphics image data representing a series of information when relating to the operation of the device corresponding to the current operation point corresponding to said current operation point in the real space The information processing system according to claim 1, further comprising: a second presentation unit that synthesizes and presents an image of an operation location of the device. The management device
Detecting means for detecting an error in the operation of the user by comparing the operation content of the user with the device and a preset operation content of the device;
A third instruction means for instructing the augmented reality presentation apparatus to synthesize and present the fact in a real space image when an error in the operation of the user by the user is detected by the detection means; Have
The augmented reality presentation device includes:
And a third presenting means for synthesizing and presenting the computer graphics image data indicating that there is an error in the operation of the user by the user in response to the instruction from the third instructing means with the video in the real space. The information processing system according to claim 1 or 2, further comprising: The management device
The information processing system according to claim 3, further comprising first recognition means for recognizing operation contents of the user for the device by analyzing image data of the body part. The management device
4. The apparatus according to claim 3, further comprising: a second recognizing unit that recognizes an operation content of the user with respect to the device based on information indicating the operation content of the user with respect to the device generated according to the operation of the user. Information processing system described in 1. The management device
The information processing system according to claim 3, further comprising third recognition means for recognizing performance contents of the user's instrument as an instrument by performing voice analysis on the user's performance sound. Executed by an augmented reality presentation device that can synthesize and display computer graphics image data on real space video, and a management device that manages time-series information related to the operation of devices in the real space Information processing method,
The augmented reality presentation device includes:
An image data generation step of generating image data of a body part on which the user operates the device;
Transmitting the image data of the body part generated by the image data generation step to the management device,
The management device
Among the time-series information related to the operation of the device, a specifying step for specifying a current operation time point in the user operation;
An acquisition step of acquiring time-series information related to the operation of the device corresponding to the current operation time point;
An identification step for identifying a position of the body part based on the image data of the body part;
The augmented reality so as to present, in the vicinity of the position of the body part identified by the identifying step, time-series information related to the operation of the device corresponding to the current operation time acquired by the acquiring step . An instruction step for instructing the presentation device,
The augmented reality presentation device includes:
In response to an instruction from the instruction unit, computer graphics image data indicating time-series information related to the operation of the device corresponding to the current operation time point is converted into an image in the vicinity of the position of the body part in the real space. An information processing method, further comprising a presentation step of combining and presenting.

Images