JP2017010228A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To select and display an annotation in accordance with a predetermined condition.SOLUTION: An information processing apparatus includes: image acquisition means of acquiring an image including a virtual object; holding means of holding annotation information indicating each of sections on the virtual object and additional information indicating a state at the time of inputting the annotation information, in association with each other; acquisition means of acquiring information input by a user and corresponding to the additional information; selection means of selecting annotation information to be displayed, on the basis of the additional information, from the annotation information for each section on the virtual object held in the holding means; generation means of generating a composite image by synthesizing the selected annotation information with the above image; and output means of outputting the composite image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program for providing mixed reality.

  In recent years, research on mixed reality (MR) for the purpose of seamless connection between a real space and a virtual space has been actively conducted. An image display device that presents mixed reality is, for example, a device having the following configuration. That is, a virtual reality space image (for example, a virtual object or character information drawn by computer graphics) generated according to the position and orientation of the imaging device on an image of the real space captured by an imaging device such as a video camera. Is a device that displays an image in which the image is superimposed and drawn. For such an apparatus, for example, an HMD (head mounted display, head mounted display) can be used.

  The image display apparatus is also realized by an optical see-through method that displays an image of a virtual reality space generated according to the position and orientation of the observer's viewpoint on an optical see-through display mounted on the observer's head. Is done.

  As an application of such an image display device, products under development in the manufacturing industry such as automobiles and airplanes are superimposed and displayed in the real world as computer graphics (CG) images, and they are confirmed and examined from a free perspective. The design etc. can be examined without making a model.

  In such an application example, since a large number of HMDs are required when a study involving a large number of people is performed, there is a system that can participate in the study using an inexpensive device such as a tablet. This system allows the person wearing the HMD to see the real-world and CG-superimposed screen displayed on the tablet screen as it is, and to participate in the study by sharing the field of view with the person wearing the HMD. Is.

  Further, in the examination using the HMD, the knowledge or memorandum obtained during the examination is stored in association with a specific part in the CG video. The information stored here is called annotation, and the HMD wearer can display the annotation character string superimposed on a specific part of the CG image superimposed in the real world and record it at the time of examination. Can be considered.

  However, when browsing a CG image associated with a plurality of annotations through the HMD, the plurality of superimposed annotations may interfere with the view of the HMD wearer.

  Therefore, the method disclosed in Patent Document 1 discloses a method in which a part in CG is selected by the pointing device held by the HMD wearer and only the annotation corresponding to the selected part is displayed.

JP 2013-225311 A

  However, in the method of Patent Document 1, only selected annotations are displayed, and thus annotations given under certain specific conditions cannot be displayed collectively.

  The present invention has been made in view of the above-described problems, and an object thereof is to select and display annotations according to specific conditions.

  In order to solve the above problems, an information processing apparatus according to the present invention includes, for example, an image acquisition unit that acquires an image including a virtual object, annotation information indicating the part for each part on the virtual object, and the annotation Holding means for associating and holding additional information indicating a state when information is input; acquisition means for receiving information corresponding to the additional information input by a user; and the virtual object held by the holding means Selection means for selecting annotation information to be displayed based on the additional information, and generation means for generating a composite image in which the selected annotation information and the image are combined, from the annotation information for each part above And an output means for outputting the composite image.

  According to the configuration of the present invention, it is possible to select and display annotations according to specific conditions.

It is a figure showing the system configuration | structure which concerns on the 1st Embodiment of this invention. It is a block diagram showing the functional structural example of the information processing apparatus which concerns on the 1st Embodiment of this invention. It is a process flowchart of the information processing apparatus which concerns on the 1st Embodiment of this invention. It is a process flowchart of the information processing apparatus which concerns on the 1st Embodiment of this invention. It is an image figure of the virtual reality image on which the annotation in the present invention was superimposed. It is an image figure of the annotation information which the annotation holding part of the information processing apparatus which concerns on the 1st Embodiment of this invention hold | maintains. It is a figure which shows the hardware structural example of the information processing apparatus of this invention.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific embodiments having the configurations described in the claims.

(First embodiment)
FIG. 1 is a diagram illustrating a system configuration using the information processing apparatus 101 according to the first embodiment of this invention.

  The information processing apparatus 101 and the HMD 104 are connected via a communication path 104. The communication path 104 is a communication path used for transmitting the video generated by the information processing apparatus 101 to the HMD 102 and for transmitting the real video acquired by the HMD 102 to the virtual reality display device 101. ) And IEEE 1394 are used. As this communication path, a data communication path such as USB or Ethernet (registered trademark) may be used, or a wireless communication protocol such as Wi-Fi may be used.

  Similarly, the information processing apparatus 101 and the tablet computer 103 are connected by a communication path 105. The communication path 105 is a communication path used for transmitting commands from the tablet computer 103 to the virtual reality display device 101 and for transmitting information related to annotations from the information processing apparatus 101 to the tablet computer 103. A wireless path such as Wi-Fi is used. A communication protocol is used. A wired connection such as USB may be used for this communication path.

  The HMD 102 is an example of a so-called head-mounted display device that is worn on the head of a user who experiences a mixed reality space that is a space in which a real space and a virtual space are fused. The HMD 102 may be a handheld handheld display.

  The tablet computer 103 accepts an annotation input from the user by an instruction input function such as touch on the CG displayed on the display unit on the tablet computer 103. Then, the annotation input is sent to the information processing apparatus 101 through the communication path. In addition, the information processing apparatus 101 has a function of displaying an image with an annotation on the display screen. The tablet computer 103 is not restricted by the shape of the tablet as long as it has a function for inputting annotations, and may be realized as a PC having a display and a keyboard, or may be annotated by voice input with a microphone. It may be a device having a function of inputting.

  FIG. 2 is a functional block diagram of the information processing apparatus 101 according to the first embodiment of this invention. As in FIG. 1, the information processing apparatus 101 is connected to an HMD 102 that is a head-mounted imaging / display device and a tablet computer 103 that is an annotation input device by wire or wirelessly. FIG. 7 is a hardware configuration diagram of the information apparatus 101 according to the present embodiment. In the figure, a CPU 710 comprehensively controls each device connected via a bus 700. The CPU 710 reads and executes processing steps and programs stored in a read-only memory (ROM) 720. In addition to the operating system (OS), each processing program, device driver, and the like according to the present embodiment are stored in the ROM 720, temporarily stored in a random access memory (RAM) 730, and appropriately executed by the CPU 710. The input I / F 740 is input as an input signal in a format that can be processed by the information processing apparatus 101 from an external device (such as a display device or an operation device). The output I / F 750 is output as an output signal to an external device (HMD 102, tablet computer 103) in a format that can be processed by the display device.

  The information processing apparatus 101 includes a head position / posture specifying unit 204, a field of view acquisition unit 205, a CG model 206, a virtual reality video generation unit 207, an annotation holding unit 208, an annotation additional information adding unit 209, an annotation narrowing unit 210, and an annotation narrowing superposition. Part 211 is provided.

  Each of these functional units is realized by the CPU 710 developing a program stored in the ROM 720 on the RAM 730 and executing processing according to each flowchart described below. Further, for example, when hardware is configured as an alternative to software processing using the CPU 710, an arithmetic unit or a circuit corresponding to the processing of each functional unit described here may be configured.

  The field-of-view acquisition unit 205 inputs an image captured by the HMD 102 to the system through a camera or the like built in the HMD 102 (image acquisition).

  The head position / posture specifying unit 204 specifies three-dimensional coordinates in the real space of the HMD 102 and its posture. In order to specify the three-dimensional coordinates and posture, information on markers and feature points existing in the field of view acquired by the field of view acquisition unit 205 may be used, an infrared marker connected to the HMD 102, a magnetic sensor, a gyro sensor, etc. May be used. Of course, information from a plurality of sensors may be used in combination.

  The CG model 206 is data representing the shape of a virtual object to be displayed while being superimposed on a real space image. This data may be data artificially designed using software such as CAD, or data reconstructed from measurement data in real space using a laser scanner or a plurality of cameras. .

  The virtual reality video generation unit 207 generates a composite video (synthesized image) in which the virtual space represented by the CG model 206 is combined with the real space video acquired by the view acquisition unit 205. In the present embodiment, the composite video generated by the virtual reality video generation unit 207 is sent to the annotation narrowing and superimposing unit 211, but may be directly output to the HMD 102 or another external device different from the HMD 102. You may display on the display. As the external display, for example, a large screen display with a fixed position can be used.

  The annotation holding unit 208 holds the annotation (annotation information) input by the tablet computer 103 in association with a specific part on the virtual object held by the CG model 206. Here, the part to which the annotation is associated may be specified using means such as a tap from the tablet, may be specified using a mouse via a PC (not shown), or the part name may be specified from the keyboard. You may specify by inputting directly.

  When a new annotation is added by the annotation holding unit 208, the annotation additional information adding unit 209 acquires some additional information and holds it in association with the annotation held in the annotation holding unit 208. As a result, the annotation holding unit 208 holds the input annotation, the specific part in the virtual object, and the additional information in association with each other. The additional information is information indicating the state of the tablet computer, the user operating the tablet computer, etc. when the annotation is input.

  In the present embodiment, the additional information is a unique ID (input device unique ID) held by the tablet computer 103 (input device) used when adding an annotation. This unique ID is an ID for identifying an individual tablet computer, and is information that specifies which tablet has added the annotation when an annotation is input using a plurality of tablet computers. Here, the unique ID may be an ID allocated in advance for each input person who inputs an annotation using a tablet, or by logging in using the ID before starting the use of the tablet or before inputting the annotation. It may be information for specifying a person who has input an annotation. For example, an ID (for example, person-specific ID information such as an employee number) indicating a user who inputs annotation information using the tablet computer 103 may be used.

  The annotation narrowing unit 210 narrows down using the additional information added by the annotation additional information adding unit 209 as a key from among the plurality of annotations held by the annotation holding unit 208, and extracts some or all of the annotations. Here, the additional information as a key may be input by the tablet computer 103, or may be input by an input device such as a mouse or a keyboard connected to the information processing apparatus 101. The wearer of the HMD 102 may input a microphone, a game controller, or the like. You may input using. Of course, these plural means may be used in combination.

  The annotation narrowing and superimposing unit 211 further superimposes (synthesizes) the annotation extracted by the annotation narrowing unit 210 on the virtual reality video generated by the virtual reality video generating unit 207, thereby generating a virtual reality video including the annotation. The video generated here is output to the HMD 102 and experienced by the wearer of the HMD 102. Further, the video may be output to a display that is different from the HMD 102 and has a fixed external position. Thereby, a user who is not wearing the HMD 102 can also observe the video.

  FIG. 5 is an image diagram showing an image generated as a result of the annotation being superimposed (synthesized) on the virtual reality image by the annotation narrowing and superimposing unit 211 in the present embodiment.

  The shape data represented by the CG model 206 is composed of four parts: a part a (501), a part b (502), a part c (503), and a part d (504). Further, the annotation holding unit 208 performs the annotation 506 for the component a (501), the annotation 507 for the component b (502), the annotation 508 for the component c (503), and the component d (504) for the above four components. Are associated with annotations 509, respectively. Here, the annotation 506 is associated with the character string “component A”, the annotation 507 is associated with the character string “component B”, the annotation 508 is associated with the character string “component C”, and the annotation 509 is associated with the character string “component D”. As a result, the character string is superimposed (synthesized) on the virtual reality image and displayed. In this figure, an object such as a signboard is projected from each component and is drawn so as to always face the HMD wearer's viewpoint. However, the annotation display form and drawing method are arbitrary. There is no limitation to the method of this embodiment.

  FIG. 3 is a flowchart expressing the behavior of the system when an annotation is input by the annotation holding unit 208 in the first embodiment of the present invention.

  First, when input of an annotation is started (S301), an annotation input person (user) inputs a component to be annotated (S302). Here, the part is a part of shape data represented by the CG model 206.

  Next, in S303, the annotation input person inputs an annotation. An annotation is an annotation stored in association with the target component specified in S302. In the present embodiment, the annotation is based on a character string, but may be an image, sound, or other media data. In the present embodiment, the annotation input person operates the tablet computer 103 held in his hand and inputs a character string. The input character string is transmitted to the information processing apparatus 101 via the network connection, and is held by the annotation holding unit 208 in association with the ID for specifying the part to be annotated. Here, the component to be assigned represents the component input in S302.

  In step S304, the annotation additional information adding unit 209 acquires the unique ID of the tablet computer 103 used for the annotation input. This unique ID is an ID for identifying an individual tablet computer. This unique ID may be an ID assigned in advance for each input person who inputs an annotation using a tablet computer, and logs in using the ID when starting the use of the tablet computer or before inputting the annotation. May be information for identifying the person who has input the annotation.

  In S305, the annotation additional information adding unit 209 holds the unique ID acquired in S304 in association with the annotation input in S303. As a result, the annotation addition target component input in S302, the annotation character string input in S303, and the additional information acquired in S304 are held in association with each other.

  In this order, the annotation input operation in this embodiment is completed (S306).

  FIG. 6 is an image of information held by the annotation holding unit 208 in the present embodiment.

  The component name 601 is a name for identifying the annotation target component. Here, the part name may be anything as long as the part to be annotated can be identified, and may be an item with low readability such as an ID uniquely assigned in the system.

  The annotation character string 602 is the character string input in S303. The information held here is superimposed and displayed on the virtual reality video as the body of the annotation. In the present embodiment, the character string information is used. However, the annotation may be video, audio, or media data combining them.

  The additional information / tablet unique ID 603 is a tablet unique ID that is additional information provided by the annotation additional information adding unit 209 in the present embodiment.

  In the present embodiment, the component name 601, the annotation character string 620, and the additional information / tablet unique ID 603 are held in association with each other. In other embodiments, when information other than a unique ID is used as additional information, the corresponding unique information is held in the column of additional information / tablet unique ID 603. Here, there may be a plurality of types of additional information. In such a case, the lines holding the additional information are spread over a plurality of lines.

  FIG. 4 is a flowchart representing the behavior of the system when a virtual reality video on which an annotation is superimposed is generated by the information processing apparatus 101 in the first embodiment of the present invention.

  First, annotation overlay display is started (S401). The superimposed display of annotation may be always displayed while the system is operating, or normally, the superimposed display of annotation may be started by the operation of the HMD wearer without displaying the annotation. The annotation refinement display function according to the present invention may be always effective during annotation display, and it is possible to arbitrarily switch between annotation non-display, all annotation display, and annotation refinement display by the operation of the wearer of the HMD 102. Also good.

  In step S <b> 402, the tablet computer 203 displays a list of annotations held by the annotation holding unit 208. This list display is a reference display before the narrowing-down condition is selected and narrowed down in S403, and S402 may be omitted. Further, it may be displayed on a display device connected to the information processing apparatus 101 instead of the tablet computer 103, or may be displayed on a display device that is not a tablet.

  In step S403, the operator of the tablet computer 103 inputs conditions for narrowing down the annotations displayed in a list. In this embodiment, the unique ID of the tablet computer used when inputting the annotation is input as a condition. Here, the condition input may be selected from options prepared in advance, a voice input using a microphone may be used as an input device, or a PC having a mouse and a keyboard may be used. Absent.

  In S404, the annotation narrowing unit 210 performs narrowing by selectively displaying only annotations having additional information that matches the condition input in S403. In the present embodiment, only annotations having a unique ID equal to the input unique ID as additional information are displayed. Here, only annotations that completely match the conditions may be selected, or all annotations that have differences within a certain range may be displayed.

  In S405, the annotation narrowing and superimposing unit 211 superimposes and displays only the annotation narrowed down in S404 on the virtual reality video. As a result, the HMD wearer can audition a virtual reality image on which only a desired annotation is displayed, and can reduce the situation where the field of view is hindered by many annotations.

  In this order, the annotation display operation in the present embodiment is finished (S406).

  According to the present embodiment, since the annotation and the ID of the user who input the annotation (or the ID of the tablet computer 103) are stored, the user can narrow down and display the annotation using the ID. Become.

(Second Embodiment)
In S304 of the first embodiment, the annotation additional information adding unit 209 acquires the date and time when the annotation is input.

  In S305, the annotation additional information adding unit 209 holds the date and time acquired in S304 in association with the annotation input in S303. As a result, the annotation addition target component input in S302, the annotation character string input in S303, and the additional information acquired in S304 are held in association with each other.

  According to the present embodiment, since the annotation and the date and time when the annotation is input are stored in association with each other, the user can narrow down and display the annotation using the specified date and time. Become.

(Third embodiment)
In S304 of the first embodiment, the annotation additional information adding unit 209 acquires the position and orientation of the tablet computer 103 in the three-dimensional space. Here, the acquisition of the position and orientation in the three-dimensional space may use information on markers and feature points existing in the field of view acquired by the camera included in the tablet computer 103, an infrared sensor connected to the tablet computer, Information such as a magnetic sensor may be used. Further, only one of the position and orientation may be acquired and the other information may not be used.

  In S305, the annotation additional information adding unit 209 holds the position and orientation in the three-dimensional space acquired in S304 in association with the annotation input in S303. As a result, the annotation addition target component input in S302, the annotation character string input in S303, and the additional information acquired in S304 are held in association with each other.

  According to the present embodiment, since the annotation and the position and orientation of the tablet computer 103 when the annotation is input are stored in association with each other, the user narrows down the annotation according to the position and orientation of the tablet computer 103. Can be displayed.

(Fourth embodiment)
In S <b> 304 of the first embodiment, the annotation additional information adding unit 209 acquires the position and orientation of the HMD 102 in the three-dimensional space when the annotation is input by the tablet computer 103. Here, the acquisition of the position and orientation in the three-dimensional space may use the function of the head position and orientation identification unit 204 provided in the information processing apparatus 101, or the position and orientation of the tablet computer 103 of the third embodiment. Other methods such as those used for acquisition may be used. Further, only one of the position and orientation may be acquired and the other information may not be used.

  In S305, the annotation additional information adding unit 209 holds the position and orientation in the three-dimensional space acquired in S304 in association with the annotation input in S303. As a result, the annotation addition target component input in S302, the annotation character string input in S303, and the additional information acquired in S304 are held in association with each other.

  According to this embodiment, since the annotation and the position and orientation of the HMD 102 when the annotation is input are stored in association with each other, the user can narrow down and display the annotation according to the position and orientation of the HMD 102. Is possible.

(Fifth embodiment)
In S304 of the first embodiment, the annotation additional information adding unit 209 acquires the position and orientation of the device held by the wearer of the HMD 102 in the three-dimensional space. Here, the device may be any device that can acquire the position and orientation. In addition, the acquisition of the position and orientation of the device in the three-dimensional space can be performed using the method used for acquiring the position of the tablet computer 103 in the third embodiment. Further, only one of the position and orientation may be acquired and the other information may not be used.

  In S305, the annotation additional information adding unit 209 holds the position in the three-dimensional space acquired in S304 in association with the annotation input in S303. As a result, the annotation addition target component input in S302, the annotation character string input in S303, and the additional information acquired in S304 are held in association with each other.

  According to the present embodiment, since the annotation and the position and orientation of the device when the annotation is input are stored in association with each other, the user can narrow down and display the annotation according to the position and orientation of the device. Is possible.

(Sixth embodiment)
In S304 of the first embodiment, the annotation additional information adding unit 209 acquires the relative positions of the HMD 102 and the devices held by the wearer of the HMD 102 in the three-dimensional space. Here, the device may be any device that can acquire the position. Further, the position of the HMD 102 and the device in the three-dimensional space can be acquired by using the method used for acquiring the position of the tablet computer 103 in the third embodiment. In addition, the attitude of the HMD 102 and the device may be acquired using the method used for acquiring the attitude of the tablet computer 103 in the third embodiment, and the relative attitude may be used as additional information in addition to the position. .

  In S305, the annotation additional information adding unit 209 holds the relative position acquired in S304 in association with the annotation input in S303. As a result, the annotation addition target component input in S302, the annotation character string input in S303, and the additional information acquired in S304 are held in association with each other.

(Seventh embodiment)
In S304 of the first embodiment, the annotation additional information adding unit 209 simultaneously acquires a plurality of additional information acquired in the third to sixth embodiments.

  In S305, the annotation additional information adding unit 209 holds the plurality of additional information acquired in S304 in association with the annotation input in S303. As a result, the annotation addition target component input in S302, the annotation character string input in S303, and the additional information acquired in S304 are held in association with each other.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (12)

Image acquisition means for acquiring an image including a virtual object;
Holding means for holding, for each part on the virtual object, annotation information indicating the part and additional information indicating a state when the annotation information is input;
Obtaining means for obtaining information corresponding to the additional information input from a user;
Selection means for selecting annotation information to be displayed from the annotation information for each part on the virtual object held in the holding means based on the information;
Generating means for generating a composite image in which the selected annotation information and the image are combined;
An information processing apparatus comprising: output means for outputting the composite image.   The information processing apparatus according to claim 1, wherein the selection unit selects annotation information associated with additional information corresponding to information input from the user.   The information processing apparatus according to claim 1, wherein the additional information is information specific to an input apparatus that inputs the annotation information.   The information processing apparatus according to claim 1, wherein the additional information is information unique to a person who has input the annotation information.   The information processing apparatus according to claim 1, wherein the additional information is information representing a date or time when the annotation information is input.   The information processing apparatus according to claim 1, wherein the additional information is a position and / or orientation of an input device used to input the annotation information when the annotation information is input.   2. The additional information is a position and / or orientation of an HMD connected to an input device used to input the annotation information when the annotation information is input. Information processing device.   The additional information is a position and / or posture of a device held by a user wearing an HMD connected to the input device used to input the annotation information when the annotation information is input. The information processing apparatus according to claim 1.   The information processing apparatus according to claim 1, further comprising an input device that inputs the annotation information. Furthermore, a head-mounted display device for the user to wear on the head is provided,
The information processing apparatus according to claim 1, wherein the output unit outputs the composite image to the head-mounted display device. An image acquisition step of acquiring an image including a virtual object;
An acquisition step of acquiring information corresponding to the additional information input from a user;
For each part on the virtual object, the annotation information indicating the part and the additional information indicating the state when the annotation information is input are stored in association with each other. A selection step of selecting annotation information to be displayed based on the additional information from the annotation information;
A generation step of generating a composite image in which the selected annotation information and the image are combined;
An information processing apparatus comprising: an output step for outputting the composite image.   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 10.

Images