JP2013093031A - Information processing apparatus, memory medium, information processing system, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve a configuration whereby data which are not displayed on a display part can be observed as a virtual object, at all times.SOLUTION: For example, in data processing applied with a PC, the data which are not displayed on a display part of the PC, e.g. data that are a target of deletion processing or copy processing, are set as a virtual object, are stuck to, for example, a finger of a hand of a user, and can be observed at all times. Accordingly, the data which are not displayed in a display area of the PC are stuck to a space, other than the display area of the PC, are displayed and can be observed so that the efficiency of the data processing is enhanced.

Description

  The present invention relates to an information processing apparatus, a storage medium, an information processing system, an information processing method, and a program. More specifically, an information processing apparatus, a storage medium, an information processing system, and an information processing method for performing data processing using mixed reality (MR) in which a real object in the real world and an electronic display are fused , As well as programs.

  For example, when a user performs data processing using a PC (personal computer), the user performs processing on display data on a display unit (display) of the PC. For example, a case where data is copied or cut (deleted) will be described as an example. In this case, the user designates a character string or an image area displayed on the display unit (display), and after the designation process, performs a process of cutting (deleting) data by inputting a command or a process of copying to another area. Do.

The processing data targeted for such cut and copy processing is stored in the memory in the PC, but may disappear from the display area in the course of processing. As a process when the user wants to confirm the contents of the process data to be cut or copied, for example, the following process is required.
(A) The processing data is always displayed in a partial area of the display unit (display).
(B) The processing data is stored in the copy buffer, and the stored data in the copy buffer is read by a user operation and displayed on the display unit (display).
(C) The processing data is displayed by being pasted (pasted) in a designated area in the display unit (display) by a user operation.

For example, if the method as described above is performed, the contents of the processing data of the cut process and the copy process can be confirmed.
However, in the above method, the method (a) requires that the processing data is always displayed on the display unit, and at least a partial area of the display unit is occupied. As a result, there arises a problem that the main work area that the user uses in the display unit, for example, the area of the document creation screen becomes narrow. Further, the method (b) requires a user operation to display the data stored in the copy buffer, which increases the processing load on the user. The method (c) also requires a user operation, and after pasting the processing data, if a mistake is found in the pasted data, new processing such as deletion processing or redo (UNDO) is performed. There is a problem that it is necessary.

  The present invention solves such a problem by using data processing utilizing, for example, mixed reality (MR). In addition, as a prior art which demonstrated mixed reality, there exist patent document 1 (Unexamined-Japanese-Patent No. 2008-304268) and patent document 2 (Unexamined-Japanese-Patent No. 2008-304269), for example. These documents describe processing for creating a real-world three-dimensional map using an image taken by a camera.

JP 2008-304268 A JP 2008-304269 A

  The present invention generates a composite image in which various data are electronically pasted in an area other than a display unit (display) such as a PC, for example, by data processing using mixed reality (MR). An information processing apparatus, a storage medium, an information processing system, an information processing method, and a program that enable effective use of a space area other than a display unit used as a main work area by enabling observation of the composite image The purpose is to provide.

The first aspect of the present invention is:
A position information acquisition unit that acquires three-dimensional position information of the finger of the user's hand included in the captured image;
An image generation unit that generates a composite image obtained by combining a finger of a user's hand and a virtual object;
The image generation unit is in an information processing apparatus that determines the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit and generates the composite image.

  Furthermore, in one embodiment of the information processing apparatus of the present invention, the image generation unit dynamically changes the virtual object display position according to the three-dimensional position information of the finger of the user's hand.

  Furthermore, in one embodiment of the information processing device of the present invention, the position information acquisition unit acquires three-dimensional position information of each of a plurality of fingers of the user, and the image generation unit acquires the position information acquisition unit. The composite image is generated by determining the display position of the virtual object based on the three-dimensional position information of each of a plurality of fingers.

  Furthermore, in one embodiment of the information processing apparatus of the present invention, the position information acquisition unit includes a three-dimensional information analysis unit that analyzes three-dimensional position information of an object included in the captured image.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the three-dimensional information analysis unit has a hand recognition module that detects an object that is determined to be a human hand or finger by image analysis.

  Furthermore, in one embodiment of the information processing apparatus of the present invention, the position information acquisition unit acquires the three-dimensional position information of the finger of the user's hand included in the captured image from the outside.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus includes a display unit, and the image generation unit outputs the generated composite image to the display unit.

  Furthermore, in an embodiment of the information processing apparatus according to the present invention, the display unit is a glasses-type display device.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus includes an imaging unit that captures the captured image.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus includes a glasses-type display device including an imaging unit and a display unit, and the image generation unit is an imaging unit of the glasses-type display device. Generates a composite image in which the virtual object is superimposed on the actual image taken by the camera and outputs the composite image to the display unit of the glasses-type display device.

  Furthermore, in an embodiment of the information processing apparatus according to the present invention, the information processing apparatus includes a communication unit, and the image generation unit generates the virtual object based on data input via the communication unit. .

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus includes a memory that stores input data via the communication unit, and the image generation unit is based on the data stored in the memory. To generate the virtual object.

  Furthermore, in one embodiment of the information processing apparatus of the present invention, the image generation unit determines the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit. The composite image is generated at the position of the finger of the hand.

  Furthermore, in one embodiment of the information processing apparatus of the present invention, the image generation unit sets the data generated or selected by a user to the virtual object and generates the composite image.

Furthermore, the second aspect of the present invention provides
An information processing system having a mixed reality generation device and a data processing device,
The data processing device includes:
Send data to the mixed reality generation device via a communication unit,
The mixed reality generation device includes:
A position information acquisition unit that acquires three-dimensional position information of the finger of the user's hand included in the captured image;
An image generation unit that generates a composite image obtained by combining a finger of a user's hand and a virtual object generated based on received data from the data processing device;
In the information processing system configured to generate the composite image by determining the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit. is there.

Furthermore, the third aspect of the present invention provides
An information processing system having a mixed reality generating device and a display device having an imaging unit and a display unit,
The display device outputs a captured image of the imaging unit to the mixed reality generation device,
The mixed reality generation device includes:
A position information acquisition unit for acquiring three-dimensional position information of a finger of a user's hand included in the captured image;
An image generation unit that generates a composite image obtained by combining a finger of a user's hand and a virtual object generated based on received data from the data processing device;
The image generation unit determines the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit, generates the composite image, and displays the display on the display device In the information processing system that outputs to the display and displays.

Furthermore, the fourth aspect of the present invention provides
A storage medium storing a program for causing an information processing apparatus to execute information processing;
Position information acquisition processing for acquiring the three-dimensional position information of the finger of the user's hand included in the captured image;
Perform image generation processing to generate a composite image that combines the finger of the user's hand and the virtual object,
In the image generation process, a storage medium storing a program for determining the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit and generating the composite image is there.

  Furthermore, in an embodiment of the storage medium of the present invention, the program is a program that dynamically changes the virtual object display position in accordance with the three-dimensional position information of the finger of the user's hand in the image generation process. .

  Furthermore, in one embodiment of the storage medium of the present invention, the program causes the position information acquisition process to acquire three-dimensional position information of each of a plurality of fingers of the user, and the position information acquisition process in the image generation process. This is a program for determining the display position of the virtual object based on the three-dimensional position information of each of the plurality of fingers acquired at the time and generating the composite image.

  Furthermore, in one embodiment of the storage medium of the present invention, the program is a program for executing an analysis process of the three-dimensional position information of the object included in the captured image in the position information acquisition process.

  Furthermore, in an embodiment of the storage medium of the present invention, the program executes processing using a hand recognition module that detects an object that is determined to be a human hand or finger by image analysis in the three-dimensional information analysis processing. It is a program to let you.

  Furthermore, in one embodiment of the storage medium of the present invention, the program is a program for acquiring, from the outside, three-dimensional position information of a finger of a user's hand included in a captured image in the position information acquisition process.

  Furthermore, in an embodiment of the storage medium of the present invention, the program is a program that causes the display unit to output a composite image generated in the image generation process.

  Furthermore, in one embodiment of the storage medium of the present invention, the program is a composite image in which the virtual object is superimposed on a real image captured by an imaging unit of a display device including an imaging unit and a display unit in the image generation process. Is generated and output to the display unit of the display device.

  Furthermore, in an embodiment of the storage medium of the present invention, the program is a program that generates the virtual object based on data input via a communication unit in the image generation processing.

  Furthermore, in an embodiment of the storage medium of the present invention, the information processing apparatus has a memory for storing input data via a communication unit, and the program stores data stored in the memory in the image generation process. Is a program for generating the virtual object based on the program.

  Furthermore, in one embodiment of the storage medium of the present invention, the program is configured to display the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired in the position information acquisition process in the image generation process. Is set to the position of the finger of the user's hand to generate the composite image.

  Furthermore, in an embodiment of the storage medium of the present invention, the program is a program for generating the composite image by setting data generated or selected by a user in the virtual object in the image generation process.

Furthermore, the fifth aspect of the present invention provides
An information processing method executed in an information processing apparatus,
A position information acquisition processing step in which the position information acquisition unit acquires the three-dimensional position information of the finger of the user's hand included in the captured image;
The image generation unit executes an image generation step of generating a composite image in which the finger of the user's hand and the virtual object are combined,
In the information processing method, the image generation step is a step of determining the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit and generating the composite image. is there.

Furthermore, the sixth aspect of the present invention provides
A program for causing an information processing apparatus to execute information processing,
A position information acquisition processing step for causing the position information acquisition unit to acquire three-dimensional position information of the finger of the user's hand included in the captured image;
Causing the image generation unit to execute an image generation step of generating a composite image obtained by combining the finger of the user's hand and the virtual object;
In the image generation step, there is a program for determining the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit and generating the composite image.

  The program of the present invention is a program that can be provided by, for example, a storage medium or a communication medium provided in a computer-readable format to an image processing apparatus or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the image processing apparatus or the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of an embodiment of the present invention, for example, in data processing to which a PC is applied, data that is not displayed on the display unit of the PC, such as data to be deleted or copied, is set as a virtual object. A structure that can be always observed by pasting on a finger or the like. According to this configuration, data that is not displayed in the display area of the PC can be pasted and displayed in a space other than the display area of the PC, and the efficiency of data processing can be increased.

It is a figure explaining one Example structure of the information processing apparatus of this invention. It is a figure explaining the example of a display of the virtual object which the information processing apparatus of this invention performs. It is a figure explaining the example of 1 process which the information processing apparatus of this invention performs. It is a figure explaining the example of a display of the virtual object which the information processing apparatus of this invention performs. It is a figure which shows the flowchart explaining the setting registration process sequence of the virtual object paste position which the information processing apparatus of this invention performs. It is a figure explaining the structural example of the table utilized in the setting registration of the virtual object paste position which the information processing apparatus of this invention performs. It is a figure which shows the flowchart explaining the virtual object sticking process sequence which the information processing apparatus of this invention performs. It is a figure which shows the flowchart explaining the data processing sequence using the virtual object corresponding data which the information processing apparatus of this invention performs. It is a figure which shows the flowchart explaining the data processing sequence using the virtual object corresponding data which the information processing apparatus of this invention performs. It is a figure explaining the example of a display of the virtual object which the information processing apparatus of this invention performs. It is a figure explaining the structural example of the table utilized in the setting registration of the virtual object paste position which the information processing apparatus of this invention performs.

  Hereinafter, details of an information processing apparatus, a storage medium, an information processing system, an information processing method, and a program according to the present invention will be described with reference to the drawings.

The present invention will be described in order according to the following items.
1. 1. Configuration example and processing example of information processing apparatus of the present invention 2. Sequence example of processing executed by information processing apparatus of the present invention Example of real object to paste virtual object

[1. Configuration example and processing example of information processing apparatus of the present invention]
A configuration example and a processing example of the information processing apparatus of the present invention will be described with reference to FIG. FIG. 1 is a diagram illustrating a configuration example of an information processing apparatus according to the present invention.

  A user 100 can perform various data processing by operating a PC (personal computer) 120. As shown in the figure, the PC 120 includes an application execution unit 121, a memory 122, and a communication unit 123. In the application execution unit 121, the application program selected by the user is executed. For example, a document creation application or a drawing application. The memory 122 includes a RAM, a ROM, etc., and is used as a storage area or work area for application programs. For example, it is also used as a storage area for processing data in copy processing and cut processing described below. The communication unit 123 performs communication processing with the mixed reality (MR) generating device 130.

  The user 100 is wearing glasses 141 having a display for displaying a virtual object. The glasses 141 are provided with a camera 142 that captures the surrounding environment. The glasses 141 and the camera 142 are connected to a mixed reality (MR) generator 130. The user 100 works while observing the display image on the display provided in the glasses 141.

  On the display of the glasses 141, a real world image that is a photographed image of the camera 142 is displayed, and a virtual object generated by the mixed reality (MR) generation device 130 is displayed together with the real world image.

  In the example of FIG. 1, the user 100 operates a PC (personal computer) 120, and the camera 142 photographs a PC (personal computer) 120 operated by the user 100. Therefore, on the display of the glasses 141, for example, an image including a display of a PC (personal computer) 120 operated by the user 100 and various real objects around it is displayed as a real world image. Further, the virtual object generated by the mixed reality (MR) generation device 130 is displayed superimposed on the real world image. Depending on the movement of the user 100, the orientation of the camera 142 is also changed. For example, when the user looks at his / her hand, an image of the hand is displayed on the display of the glasses 141 as a real world image, and a virtual object is displayed on the real world image. It will also be displayed. For example, a composite image of a real object and a virtual object as shown in FIG.

  Prior to the description of FIG. 2, the configuration of the mixed reality (MR) generation device 130 shown in FIG. 1 will be described. As shown in FIG. 1, the mixed reality (MR) generation device 130 includes a three-dimensional information analysis unit 131, a virtual object management module 132, a memory 133, and a communication unit 134.

  The three-dimensional information analysis unit 131 inputs a photographed image of the camera 142 worn by the user and performs a process of analyzing the three-dimensional position of the object included in the photographed image. This three-dimensional position analysis process is performed as a process to which, for example, SLAM (simultaneous localization and mapping) is applied. SLAM is a process of selecting a feature point from various real objects included in a captured image of a camera and detecting the position of the selected feature point and the position and orientation of the camera together. Note that SLAM is described in Patent Document 1 (Japanese Patent Laid-Open No. 2008-304268) and Patent Document 2 (Japanese Patent Laid-Open No. 2008-304269). The basic processing of SLAM is described in the paper [Andrew J. Davison, “Real-time simultaneous localization and mapping with a single camera”, Proceedings of the 9th International Conference on Computer Vision, 3] (200).

  The three-dimensional information analysis unit 131 calculates the three-dimensional position of the real object included in the photographed image of the camera 142 worn by the user by applying the above SLAM, for example. However, the three-dimensional information analysis unit 131 is not limited to the above-described SLAM, and may be set to obtain the three-dimensional position of the object included in the camera captured image by other methods.

  The virtual object management module 132 manages virtual objects displayed on the display of the glasses 141 worn by the user. The virtual object is data stored in the memory 133. Specifically, for example, the display display image 200 shown in FIG. 2 is displayed on the display of the glasses 141 worn by the user. The hand image included in the display display image 200 is a real image (real object) taken by the camera 142. Virtual objects 201 to 203 shown in FIG. 2 are displayed together with this real image (real object).

  The user 100 illustrated in FIG. 1 can observe an image in which the virtual object 150 illustrated in FIG. A virtual object 150 illustrated in FIG. 1 corresponds to the virtual objects 201 to 203 illustrated in FIG. 2 and is not a real-world object (real object).

  Virtual objects 201 to 203 shown in FIG. 2 that are data corresponding to the virtual object 150 shown in FIG. 1 are processing data that has been subjected to copy processing and cut processing by the operation of the PC 120 by the user 100. This processing data is stored in the memory 122 in the PC 120 by the processing of the application execution unit 121 of the PC 120 according to the operation of the user 100 on the PC 120.

  The data stored in the memory 122 is transmitted to the mixed reality (MR) generation device 130 through communication between the communication unit 123 of the PC 120 and the communication unit 134 of the mixed reality (MR) generation device 130. The mixed reality (MR) generation device 130 stores the received data from the PC 120 in the memory 133 in the mixed reality (MR) generation device 130.

  The virtual object management module 132 of the mixed reality (MR) generation device 130 performs processing for displaying the data stored in the memory 133 as a virtual object on the display of the glasses 141 worn by the user. Specifically, for example, a display position of the virtual object is determined, and a process of displaying the virtual object at the determined display position is performed.

In the example illustrated in FIG. 2, three virtual objects 201 to 203 are displayed with the display positions set on the fingertips of the respective fingers of the user's [hand] displayed as a real world image.
Virtual object 201 [Change],
Virtual object 202 [Yes, we can],
Virtual object 203 [photograph image],
These are all data stored in the memory 122 of the PC 120 as data to be cut or copied by the operation of the PC 120 by the user 100. These data are transmitted to the mixed reality (MR) generator 130 by communication and stored in the memory 133 of the mixed reality (MR) generator 130. The virtual object management module 132 of the mixed reality (MR) generation device 130 performs processing for determining the display position of each virtual object using the data stored in the memory 133 as a virtual object, and displays the display of the glasses 141 worn by the user. To display.

  In the apparatus shown in FIG. 1, the PC 120 and the mixed reality (MR) generation apparatus 130 are used as separate apparatuses for data communication between the two apparatuses. However, the PC 120 and the mixed reality ( The MR) generation device 130 can be configured as one device. In this case, the communication unit of each device shown in FIG. 1 can be omitted. For example, the three-dimensional information analysis unit 131 and the virtual object management module 132 of the mixed reality (MR) generation device 130 may be set in the PC 120 to perform processing in one PC.

  In other words, FIG. 1 shows a PC (personal computer) 120 and a mixed reality (MR) generator 130, but the information processing apparatus of the present invention is a mixed reality (MR :) shown in FIG. A mixed reality (MR) generation device 130 may be a single device or a mixed reality (MR) generation device 130 and a PC 120. Or it can also be comprised by one apparatus which has the function of two apparatuses together.

  A specific processing example to which the configuration of FIG. 1 is applied will be described with reference to FIG. For example, the user 100 executes a document creation application in the application execution unit 121 of the PC 120 to create a document as shown in FIG. Here, when the data area 301 is set and a cut command is input, the data in the data area 301 is stored in the memory 122 of the PC 120. Thereafter, this data is transmitted to the mixed reality (MR) generation device 130 by communication processing and stored in the memory 133 in the mixed reality (MR) generation device 130.

  The three-dimensional information analysis unit 131 of the mixed reality (MR) generation device 130 inputs a photographed image of the camera 142 attached to the user 100 and calculates the three-dimensional position of the object included in the photographed image by applying, for example, SLAM. To do. For example, markers 321a to 321d set at the four corners of the PC display shown in FIG. 3 are selected as feature points, and the three-dimensional positions of these feature points are calculated.

  The example shown in FIG. 3 shows an example in which the markers 321a to 321d are set for identifying the feature points. However, it is not essential to set such markers. It is also possible to detect a corner (corner) of an object, select the detected corner (corner) as a feature point, and determine the three-dimensional position of the feature point. The corner detection process can be performed by a feature point extraction process using an existing Harris Corner Detector.

  Note that the three-dimensional information analysis unit 131 may have a function of detecting a specific object from a captured image. Specifically, a configuration in which a hand recognition module for recognizing a human hand or finger can be provided. The hand recognition module is a module that detects an object that is determined to be a human hand or finger by analyzing an image. Such a specific object detection module retains characteristic information of a specific object (for example, a human hand or a finger), and determines an detected object as a specific object when an object having the characteristic information is detected from an image. I do.

  The following documents disclose the finger recognition algorithm. [Handy AR: Markerless Inspection of Augmented Reality Objects Using Finger Tracking]. For example, the three-dimensional information analysis unit 131 is configured to include a module that executes an algorithm described in this document, and configured to execute processing for recognizing a human hand or finger.

  By calculating the three-dimensional position of each finger of the user's hand from the image taken by the camera 142 and pasting and displaying the virtual object at that position, the display display image 200 described above with reference to FIG. Can be generated.

  The virtual object management module 132 acquires the 3D position information of the real object included in the camera captured image from the 3D information analysis unit 131. For example, the position information of each finger of the user is acquired. Further, the virtual object management module 132 acquires copy data and cut data stored in the memory 133, and uses the acquired data as a virtual object as an image pasted on the position of each finger of the user 100. A process of displaying on the display 141 is performed.

  FIG. 4 shows an example of data displayed on the display of the glasses 141 worn by the user 100 when the data area 301 shown in FIG. 3 is cut by the user and set as a virtual object. A display display image 350 illustrated in FIG. 4 is an image displayed on the display of the glasses 141 worn by the user 100.

  The images on the display in which the markers 321a to 321d are set are actual images of the PC 120 operated by the user 100. It is the real image which the camera 142 image | photographed. The hand 360 is also an image of the user's actual hand. A virtual object 371 is displayed together with this real image. The virtual object 371 is data to be processed in the process for the PC 120 by the user 100, for example, data that has been cut or copied.

  The virtual object 371 is data stored in the memory 133 of the mixed reality (MR) generation device 130. The virtual object management module 132 acquires the three-dimensional position information of the real object (the finger of the user's hand) included in the camera-captured image from the three-dimensional information analysis unit 131, and performs virtual processing at a position corresponding to the acquired position of the real object. The display position of the object 371 is set and display processing is executed.

  Through these processes, the user 100 can always confirm the data that has been deleted from the display area of the PC 120 as the virtual object 371.

[2. Sequence example of processing executed by information processing apparatus of present invention]
Next, a processing sequence such as processing for setting each finger of the user as a virtual object pasting display position and processing for displaying a virtual object will be described with reference to the flowchart in FIG. The processing sequence described below is as follows.
(A) Processing for setting and registering each finger of the user as a virtual object pasting display position (FIG. 5)
(B) Processing for pasting and displaying a virtual object on each finger of the user (FIG. 7)
(C) Processing of the mixed reality (MR) generation device 130 when using (pasting) the virtual object set on the user's finger as processing data in the PC (FIG. 8)
(D) Processing of PC 120 when using (pasting) the virtual object set on the user's finger as processing data on the PC (FIG. 9)

(A) Process for Setting and Registering Each Finger of User as Virtual Object Pasting Display Position First, a process for setting and registering each finger of a user as a virtual object pasting display position will be described with reference to the flowchart shown in FIG. This processing is executed as processing on the mixed reality (MR) generation device 130 side.

  First, in step S101, the mixed reality (MR) generation device 130 is set to the paste destination registration processing mode. Although not shown in the mixed reality (MR) generation device 130 shown in FIG. 1, the mixed reality (MR) generation device 130 has an input unit for mode setting and the like, and the mode can be set by a user operation. It is.

  Next, in step S102, the three-dimensional information analysis unit (hand recognition module) 131 analyzes the input image from the camera 142, and analyzes the three-dimensional position of the real object included in the image. In this example, the real object to be analyzed for the three-dimensional position is each finger of the user's hand. The three-dimensional information analysis unit (hand recognition module) 131 identifies each finger of one hand of the user and analyzes the three-dimensional position of each finger.

  In step S103, when the user instructs (touches) the finger to be pasted with the finger of the other hand, the three-dimensional information analysis unit (hand recognition module) 131 determines which one of the fingers by the analysis of the image in step S104. Recognize whether a finger is pointed. The three-dimensional information analysis unit (hand recognition module) 131 has human finger identification information in advance, and can identify whether the finger designated by the user is a thumb or a finger. The three-dimensional information analysis unit (hand recognition module) 131 outputs information on the finger designated by the user as the virtual object paste position to the virtual object management module 132.

  Next, in step S <b> 105, the virtual object management module 132 registers the finger specified as the virtual object paste position input from the three-dimensional information analysis unit (hand recognition module) 131. Note that if the designated finger has already been registered as the virtual object paste position, the user is notified that it has been registered. Ignore without processing.

  The virtual object management module 132 includes, as management information, position information registered as a virtual object paste position and state information indicating whether or not the virtual object has already been pasted at the paste position. The table is held in the memory 133 and managed. For example, it is a management table storing information as shown in FIG. The virtual object management module 132 performs processing while appropriately updating and referring to such a virtual object information management table.

  In step S106, it is determined whether or not the pasting destination registration processing mode is continuing. If the pasting registration processing mode is continuing, the processing from step S102 onward is repeatedly executed. In step S106, if it is determined that the pasting destination registration processing mode is not continuing, the registration processing ends.

  The process in FIG. 5 is a process in which the virtual object paste destination is set to the finger of the hand, but any other object can be set as the virtual object paste destination. For example, as will be described later, as shown in FIG. 10, it is also possible to perform processing such as setting a PC frame as a virtual object paste position.

  Next, a virtual object pasting processing sequence using the virtual object pasting destination registered according to the flow shown in FIG. 5 will be described with reference to the flowchart shown in FIG. The flow shown in FIG. 7 is a process of pasting a virtual object on each finger of the user and displaying it on the display of the glasses 141 worn by the user 100. This process is executed as a process on the mixed reality (MR) generation apparatus 130 side shown in FIG.

  First, in step S <b> 151, the virtual object management module 132 of the mixed reality (MR) generation device 130 acquires data stored in the memory 133 of the mixed reality (MR) generation device 130. Note that this data is processing data received from the PC 120 via the communication unit 134 under the control of the control unit of the mixed reality (MR) generation device 130. That is, it is data that has been cut or copied by user data processing on the PC 120.

  Next, in step S152, the virtual object management module 132 checks the presence / absence of a pastable position where a virtual object is not pasted. This processing may be executed by the 3D information analysis unit 131 using analysis information based on the acquired image of the camera 142, or by referring to the virtual object information management table described above with reference to FIG. It is good also as a structure to perform.

  In step S153, it is determined whether the virtual object is pasted at all registered virtual object pasting positions. If virtual objects are pasted at all registered virtual object pasting positions, the determination in step S153 is Yes, and the process proceeds to step S156. On the other hand, when the virtual object pasting position remains, the determination in step S153 is No, and the process proceeds to step S154.

  For example, when referring to the virtual object information management table in which the registration information shown in FIG. 6 is set, entries (3) and after are detected as virtual object pasting positions where no virtual object is pasted at the present time. become. In this case, since the virtual object pasting position remains, the determination in step S153 is No, and the process proceeds to step S154.

  In step S154, the paste destination is determined in the registration order of the available paste positions (for example, thumb, index finger, middle finger). The setting of the pasting position may be configured to be selectable by the user.

  Next, proceeding to step S155, the virtual object management module 132 executes the process of pasting the data acquired from the memory at the previous step S151 at the selected virtual object paste position.

  The three-dimensional information analysis unit 131 analyzes the input image from the camera 142 as needed, and grasps the three-dimensional position of each finger of the hand that is a real object. The virtual object management module 132 attaches the composite image of the real object and the virtual object to the user 100 while dynamically changing the display position of the virtual object according to the finger position information input from the three-dimensional information analysis unit 131. Processing to display on the display of the glasses 141 is performed.

  By this process, for example, the virtual object pasting display data described with reference to FIGS. 2 and 4 is displayed on the display of the glasses 141 worn by the user.

  Next, a process when the determination of Yes is made in step S153 will be described. In this case, a virtual object is pasted at all registered virtual object pasting positions, the determination in step S153 is Yes, and the process proceeds to step S156.

  In step S156, an inquiry is made to the user. That is, an inquiry is made to the user as to whether or not the virtual object can be deleted. For this inquiry, for example, a message is transmitted to the PC 120 operated by the user via the communication unit and displayed on the display of the PC. Alternatively, the message is output using the output unit (speaker or display) of the mixed reality (MR) generation device 130. Note that the selection process of the virtual object to be deleted may be, for example, a process and a setting for inquiring the user whether deletion is possible sequentially from the virtual object on which the oldest pasting process has been performed. Or it is good also as a structure which performs the process which designates the virtual object which the user judged to be able to delete from the some virtual object pasted, and selects the deletion object by the designation | designated information.

  If the input of deletion OK from the user is obtained, the process proceeds to step S157 to delete the virtual object. The user input may be performed via the PC 120 or via the input unit of the mixed reality (MR) generation device 130. Alternatively, analysis based on the image of the camera 142 acquired by the three-dimensional information analysis unit 131 of the mixed reality (MR) generation device 130, for example, using the designation information of the virtual object that can be deleted designated by the user may be executed. Good.

  When this deletion process is performed, the process proceeds to step S155. In step S155, the virtual object management module 132 executes the process of pasting data acquired from the memory in step S151 in the virtual object pasting position set by the deletion process. By this process, for example, the virtual object pasting display data described with reference to FIGS. 2 and 4 is displayed on the display of the glasses 141 worn by the user.

  On the other hand, if the deletion OK input from the user is not obtained in step S156, the process proceeds to step S158. In step S158, processing for notifying the user that the new lending object paste display processing cannot be executed is performed, and the processing is terminated. In this user notification process, for example, a message is transmitted to the PC 120 operated by the user via the communication unit and displayed on the display of the PC. Alternatively, the message is output using the output unit (speaker or display) of the mixed reality (MR) generation device 130.

  Next, a processing sequence for using (pasting) a virtual object as processing data in a PC will be described with reference to flowcharts shown in FIGS. This process is performed by a process on the mixed reality (MR) generation apparatus 130 side and a process on the PC 120 side.

  First, processing of the mixed reality (MR) generation device 130 will be described with reference to a flowchart shown in FIG. First, in step S201, the mixed reality (MR) generation device 130 analyzes instruction information on a virtual object pasting position (for example, a finger F).

  While observing the display of the glasses 141, the user 100 designates one virtual object with the finger of the other hand from a plurality of virtual objects pasted on the user's finger as shown in FIG. A three-dimensional information analysis unit (hand recognition module) 131 analyzes an input image from the camera 142 and analyzes a position indicated by a user included in the image and a finger that is a real object at the position. . This information is provided to the virtual object management module 132.

  Next, in step S202, the virtual object management module 132 determines whether there is a virtual object pasted at the user's designated position (for example, one of the user's fingers) analyzed by the three-dimensional information analysis unit (hand recognition module) 131. Determine whether or not. This processing can be performed with reference to, for example, the virtual object information management table described with reference to FIG.

If a virtual object is pasted at the user's designated position, the determination in step S203 is Yes and the process proceeds to step S204.
In step S <b> 204, data corresponding to the virtual object pasted at the user's designated position is acquired from the memory 133 and transmitted to the PC 120 via the communication unit 134.

  The virtual object management module 132 performs data processing using the PC 120 on data corresponding to the virtual object requested to be processed in response to the input of the processing request for the virtual object displayed on the display of the glasses 141 of the user 100. A process of outputting to an accessible memory of the processing device or the PC 120 is performed.

  In the configuration shown in FIG. 1, the mixed reality (MR) generation device 130 outputs data to the PC 120 side via the communication unit 134. The PC 120 receives data from the mixed reality (MR) generation device 130 side and stores it in the memory 122 on the PC 120 side.

On the other hand, if the virtual object is pasted at the user's designated position, the determination in step S203 is No, and the process proceeds to step S205.
In step S205, it is notified that the virtual object is not pasted at the user's designated position. This notification is performed via the output unit of the PC 120 or the mixed reality (MR) generation device 130.

  Next, a sequence of processing executed on the PC 120 side after completion of processing on the mixed reality (MR) generation device 130 side shown in FIG. 8 will be described with reference to a flowchart shown in FIG. The PC 120 receives data as processing target data from the mixed reality (MR) generation device 130 side and stores the data in the memory 122 on the PC side.

  First, in step S301, the PC 120 inputs screen position instruction information from the user. This is the position where the data corresponding to the virtual object is pasted. For example, it is specified by operating a mouse or a keyboard. Alternatively, in the case of a touch display, the touch position information is calculated by touching the screen with the user's finger.

In step S302, the application execution unit 121 of the PC 120 calculates position coordinates (x, y) on the display instructed by the user.
In step S303, the location corresponding to the designated position (x, y) is determined as the data insertion destination. For example, between characters.

  In step S304, it is determined whether or not the processing target data received from the mixed reality (MR) generation device 130 side exists in the memory 122 in the PC 120. If processing target data exists, the determination in step S304 is Yes, and in step S305, the application execution unit 121 extracts the data from the memory 122 and executes data paste at the determined insertion destination. If there is no data to be processed, the determination in step S304 is No, and the process ends without performing the process in step S305.

  In this way, the virtual object displayed on the glasses 141 worn by the user 100 is used as processing data of the PC 120 operated by the user in the real world.

[3. Example of real object to paste virtual object]
In the above-described embodiment, the processing example using the finger of the user's hand as the virtual object paste destination has been described. However, the virtual object paste destination is not limited to the finger of the user's hand, but the glasses worn by the user 100 Various real objects displayed in 141 can be used.

  For example, as shown in FIG. 10, processing such as setting the outer frame of the PC 120 as the virtual object paste position is also possible. FIG. 10 shows a display image 350 displayed on the display of the glasses 141 worn by the user 100, as described with reference to FIG.

  The images on the display in which the markers 321a to 321d are set are real images of the PC 120 operated by the user 100, and are real images taken by the camera 142. Virtual objects 401 to 403 are displayed together with this real image. The virtual objects 401 to 403 are data to be processed in the process for the PC 120 by the user 100, for example, data that has been cut or copied.

  In the example illustrated in FIG. 10, the pasting positions of these virtual objects 401 to 403 are the section between the marker 321 a and the marker 321 b on the right side of the PC 321. That is, a virtual object pasting section 380 indicated by an arrow in FIG. 10 is set. After obtaining the coordinates (xa, ya, za) in the real space of the marker 321a and the coordinates (xb, yb, zb) in the real space of the marker 321b, this interval (xa, ya, za) to (xb, yb, zb) is obtained. ) Are equally divided into a plurality of sections, and each divided area is set as a virtual object pasting area.

  The virtual object information management table in this case is set as shown in FIG. 11, for example. The table shown in FIG. 11 also shows an actual pasted virtual object as reference data. As described above, various objects can be used for the virtual object pasting position without the user's hand.

  According to the configuration of the present invention, the copied content can be confirmed without narrowing the information presentation screen, displaying the copy buffer one by one, or pasting and UNDOing once. Also, if you use your finger or your body as a copy buffer, when you paste the copied content from one information presentation screen to another information presentation screen at a remote location, you move to that screen, Since pasting is possible, the pasting operation can be performed intuitively. In addition, since the 3D object is instantiated and displayed as a 3D object in the real world, it is easy to intuitively understand its shape and the direction and the like when pasting.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the embodiment of the present invention, for example, in data processing using a PC, data that is not displayed on the display unit of the PC, such as data targeted for deletion processing or copy processing, is displayed as a virtual object. It was set as, and it was set as the structure which can affix on a user's finger etc. and can always observe. According to this configuration, data that is not displayed in the display area of the PC can be pasted and displayed in a space other than the display area of the PC, and the efficiency of data processing can be increased.

100 users 120 PC (personal computer)
DESCRIPTION OF SYMBOLS 121 Application execution part 122 Memory 123 Communication part 130 Mixed reality (MR) production | generation apparatus 131 Three-dimensional information analysis part 132 Virtual object management module 133 Memory 134 Communication part 141 Glasses 142 Camera 150 Virtual object 200 Display display image 201-203 Virtual object 321a to 321d Marker 350 Display display image 360 Hand 371 Virtual object 380 Virtual object pasting section 401 to 403 Virtual object

Claims (30)

A position information acquisition unit that acquires three-dimensional position information of the finger of the user's hand included in the captured image;
An image generation unit that generates a composite image obtained by combining a finger of a user's hand and a virtual object;
The information processing apparatus, wherein the image generation unit determines the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit and generates the composite image.   The information processing apparatus according to claim 1, wherein the image generation unit dynamically changes the virtual object display position according to three-dimensional position information of a finger of a user's hand. The position information acquisition unit acquires three-dimensional position information of each of a plurality of fingers of the user,
The information processing according to claim 1, wherein the image generation unit determines the display position of the virtual object based on the three-dimensional position information of each of the plurality of fingers acquired by the position information acquisition unit and generates the composite image. apparatus.   The information processing apparatus according to claim 1, wherein the position information acquisition unit includes a three-dimensional information analysis unit that analyzes three-dimensional position information of an object included in a captured image.   The information processing apparatus according to claim 4, wherein the three-dimensional information analysis unit includes a hand recognition module that detects an object that is determined to be a human hand or a finger by image analysis.   The information processing apparatus according to claim 1, wherein the position information acquisition unit acquires three-dimensional position information of a finger of a user's hand included in the captured image from the outside. The information processing apparatus has a display unit,
The information processing apparatus according to claim 1, wherein the image generation unit outputs the generated composite image to the display unit.   The information processing apparatus according to claim 7, wherein the display unit is a glasses-type display device.   The information processing apparatus according to claim 1, further comprising an imaging unit that captures the captured image. The information processing apparatus includes a glasses-type display device including an imaging unit and a display unit,
The said image generation part produces | generates the synthesized image which superimposed the said virtual object on the real image image | photographed by the imaging part of the said glasses-type display apparatus, and outputs it to the display part of the said glasses-type display apparatus. Information processing device. The information processing apparatus includes a communication unit,
The information processing apparatus according to claim 1, wherein the image generation unit generates the virtual object based on data input via the communication unit. The information processing apparatus has a memory for storing input data via the communication unit,
The information processing apparatus according to claim 11, wherein the image generation unit generates the virtual object based on data stored in the memory.   The image generation unit sets the display position of the virtual object to the position of the finger of the user's hand based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit, and generates the composite image The information processing apparatus according to claim 1.   The information processing apparatus according to claim 1, wherein the image generation unit sets the data generated or selected by a user to the virtual object and generates the composite image. An information processing system having a mixed reality generation device and a data processing device,
The data processing device includes:
Send data to the mixed reality generation device via a communication unit,
The mixed reality generation device includes:
A position information acquisition unit that acquires three-dimensional position information of the finger of the user's hand included in the captured image;
An image generation unit that generates a composite image obtained by combining a finger of a user's hand and a virtual object generated based on received data from the data processing device;
The information processing system, wherein the image generation unit determines the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit and generates the composite image. An information processing system having a mixed reality generating device and a display device having an imaging unit and a display unit,
The display device outputs a captured image of the imaging unit to the mixed reality generation device,
The mixed reality generation device includes:
A position information acquisition unit for acquiring three-dimensional position information of a finger of a user's hand included in the captured image;
An image generation unit that generates a composite image obtained by combining a finger of a user's hand and a virtual object generated based on received data from the data processing device;
The image generation unit determines the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit, generates the composite image, and displays the display on the display device An information processing system that outputs and displays on the screen. A storage medium storing a program for causing an information processing apparatus to execute information processing;
Position information acquisition processing for acquiring the three-dimensional position information of the finger of the user's hand included in the captured image;
Perform image generation processing to generate a composite image that combines the finger of the user's hand and the virtual object,
A storage medium storing a program for generating a composite image by determining a display position of the virtual object based on three-dimensional position information of a finger of a user's hand acquired by the position information acquisition unit in the image generation processing.   The storage medium according to claim 17, wherein the program is a program that dynamically changes the virtual object display position according to three-dimensional position information of a finger of a user's hand in the image generation process. In the position information acquisition process, the program acquires three-dimensional position information of each of the user's fingers,
18. The program according to claim 17, wherein the image generation process determines a display position of the virtual object based on three-dimensional position information of each of a plurality of fingers acquired in the position information acquisition process and generates the composite image. Storage media.   The storage medium according to claim 17, wherein the program is a program for executing an analysis process of three-dimensional position information of an object included in a captured image in the position information acquisition process.   The storage medium according to claim 20, wherein the program is a program that executes processing using a hand recognition module that detects an object that is determined to be a human hand or finger by image analysis in the three-dimensional information analysis processing.   The storage medium according to claim 17, wherein the program is a program for acquiring three-dimensional position information of a finger of a user's hand included in a captured image from the outside in the position information acquisition process.   The storage medium according to claim 17, wherein the program is a program that causes the display unit to output a composite image generated in the image generation process.   In the image generation process, the program generates a composite image in which the virtual object is superimposed on a real image captured by an imaging unit of a display device including an imaging unit and a display unit, and outputs the composite image to the display unit of the display device The storage medium according to claim 17, which is a program to be executed.   The storage medium according to claim 17, wherein the program is a program for generating the virtual object based on data input via a communication unit in the image generation process. The information processing apparatus has a memory for storing input data via a communication unit,
The storage medium according to claim 17, wherein the program is a program for generating the virtual object based on data stored in the memory in the image generation processing.   In the image generation process, the program sets the display position of the virtual object to the position of the finger of the user's hand based on the three-dimensional position information of the finger of the user's hand acquired in the position information acquisition process. The storage medium according to claim 17, wherein the storage medium is a program for generating a composite image.   The storage medium according to claim 17, wherein the program is a program that sets data generated or selected by a user in the virtual object in the image generation process to generate the composite image. An information processing method executed in an information processing apparatus,
A position information acquisition processing step in which the position information acquisition unit acquires the three-dimensional position information of the finger of the user's hand included in the captured image;
The image generation unit executes an image generation step of generating a composite image in which the finger of the user's hand and the virtual object are combined,
The image generation step is an information processing method which is a step of determining the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit and generating the composite image. A program for causing an information processing apparatus to execute information processing,
A position information acquisition processing step for causing the position information acquisition unit to acquire three-dimensional position information of the finger of the user's hand included in the captured image;
Causing the image generation unit to execute an image generation step of generating a composite image obtained by combining the finger of the user's hand and the virtual object;
In the image generation step, a program for determining the display position of the virtual object based on the three-dimensional position information of the finger of the user's hand acquired by the position information acquisition unit and generating the composite image.