JP2015087909A - Information processing system, information processing device, information processing server, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To have a person to be viewed displayed and have a person unnecessary to be viewed non-displayed regarding an occlusion problem with an augmented reality (AR) content in an augmented reality space and a person in an actual space.SOLUTION: A CPU 104 is configured to control a display of digital image information (an AR content) deleting an image in a region overlapping a display in an augmented reality space of a browse user having a right to browse the AR content and an actual space in a display screen of a display 101 provided in an HMD 100. In the augmented reality space, the display on the display screen is configured to be controlled so that the browse user is located ahead the digital image information and a non-browse user having no right to browse the AR content is located at a deeper location than the digital image information.

Description

  The present invention relates to an information processing system, an information processing apparatus, an information processing server, an information processing method, and a program. Specifically, the present invention relates to an information processing system, an information processing apparatus, an information processing server, an information processing method, and a program that perform composite display on a mobile display device or the like using augmented reality display technology.

  When displaying a certain content as AR content, the way the content is viewed from the positional relationship between the position of the AR viewer and the position where the AR content is displayed so that it appears as if it exists in the real world. There is a technique for calculating and displaying AR content. As such AR content, for example, when a document is displayed, it is possible to make it appear as if there is a whiteboard. In such a use case where the AR content is viewed by multiple people, the AR viewer makes the AR content available to other AR viewers in the same way that someone points to the content written on the whiteboard. It is possible to point with a finger. However, at this time, there arises an occlusion problem that an AR viewer who wants to be seen in front of the AR content is hidden in the displayed AR content.

  In response to this problem, the occlusion problem that occurs between the real world object and the AR content can be reduced by changing the composition control of the AR content based on the viewpoint position of the AR viewer and the position of the real world object. There is a solution (for example, Patent Document 1). In addition, when AR content hides an object in the real world, it is possible to solve the occlusion problem that occurs between the object in the real world and the AR content by making the hidden AR content transparent. There is (for example, Patent Document 2).

JP 2005-157610 A JP 2005-128877 A

Tomoka Nakagawa, Tomokazu Sato, Naokazu Yokoya: "Camera position and orientation estimation from still images by voting using landmark database" Information Processing Society of Japan Research Report CVIM152-13, Jan. 2006. Koichi Sagawa, Mitsutoshi Son, Hikaru Tsujioka: “Unconstrained measurement of three-dimensional walking movement by acceleration integration” The 202nd meeting of the Society of Instrument and Control Engineers Tohoku Branch (2002/7/2) Document No. 202-10 Tomoya Ishikawa, Masakatsu Oki, Takashi Otsuki, Takeshi Kurata: “Indoor Pedestrian Tracking System Based on Synergistic Service Collaboration” The Institute of Electronics, Information and Communication Engineers Vol.109, No.373, pp.249-254 (2009) http://s.webry.info/sp/kawa.at.webry.info/200905/article_14.html http://www.ted.com/talks/lang/en/pranav_mistry_the_thrilling_potential_of_sixthsense_technology.html http://www.jsme.or.jp/monograph/rmd/2000/PDF/1P1-68/1P1-68-104.PDF Richard A. Newcombe, Steven J. Lovegrove and Andrew J. Davison: “DTAM: Dense Tracking and Mapping in Real-Time” ICCV 2011

  Mainly from the security aspect, a viewer who has a right to view a certain AR content (AR-X) (AR-X viewer) can see the AR content X, but other people (AR-X non-viewers) It is conceivable to control the AR display so that the AR content X is not visible to the user. At that time, the AR-X non-browser does not notice the display of the AR content visible to the AR-X viewer, and interrupts between the AR-X viewer and the place where the AR content is displayed. Therefore, the AR-X viewer's field of view may be obstructed.

  In order to solve such a problem, if the person who interrupted between the AR-X viewer and the AR content is an AR-X viewer, the AR-X viewer is displayed, and the AR-X non-viewer is displayed. Although it is desired to prevent display if there is, such control of display cannot be realized by the conventional technology.

  An information processing system according to the present invention is an information processing system including a display device that displays real space and digital image information on a display screen and presents an augmented reality space on the display screen, and the display device in the real space First acquisition means for acquiring first information indicating the presence position of the display device; second information for acquiring whether or not a user wearing the display device is set to be able to view the digital image information; 2 acquisition means, third acquisition means for acquiring third information indicating the position and size for display of the digital image information in the augmented reality space, based on the first to third information, Browsing that is set to allow browsing of the digital image information that exists between the position in the real space corresponding to the position for display and the presence position for each user A determination means for determining a user and a non-browsing user who is not set, and displaying the digital image information and the real space in which the image of the region overlapping the display in the augmented reality space of the browsing user is displayed on the display screen Thus, in the augmented reality space, control means for controlling display on the display screen so that the browsing user is positioned in front of the digital image information and the non-viewing user is positioned in the back of the digital image information. It is characterized by providing.

  Regarding the digital image information and the occlusion problem of the user in the real space, it is possible to realize a control operation in which a viewing user who should be seen is displayed on the display screen and a non-viewing user who does not need to see is not displayed on the display screen.

It is a hardware block diagram of HMD100 used by an embodiment of the present invention. It is a figure which shows the external appearance of HMD100 used by embodiment of this invention. It is a hardware block diagram of a processing server used in an embodiment of the present invention. It is a figure which shows the system configuration | structure of embodiment of this invention. It is the figure shown as what looked at the example of the positional relationship of the user and AR content in Embodiment 1 from the top. It is a figure which demonstrates concretely the image | video which the AR viewer A in Embodiment 1 sees through HMD. 6 is a flowchart illustrating a process flow in a user location registration process according to the first embodiment. 6 is a diagram illustrating an example of a personal attribute information management table held on the HDD by the processing server in the first embodiment. FIG. 6 is a flowchart illustrating a process flow in an AR projection position determination process according to the first embodiment. FIG. 6 is a diagram illustrating an example of an AR content location management table held on a HDD by a processing server in the first embodiment. 6 is a flowchart illustrating a process flow in an AR projection process according to the first embodiment. 12 is a flowchart showing a process flow in a display display process executed in S1112 of FIG. 11 according to the first embodiment. It is a figure which shows the AR viewer's visual field in Embodiment 1. FIG. It is the figure which showed that the direction of the human body contained in the image | video image | photographed with the camera with a built-in HMD with respect to AR viewer in Embodiment 1 was seen from the top. It is the figure shown as what looked at the example of the positional relationship of the user and AR content in Embodiment 2 from the top. It is a figure which demonstrates concretely the image | video which the AR viewer A in Embodiment 2 sees through HMD. 12 is a flowchart showing a flow of display display processing executed in S1112 of FIG. 11 according to the second embodiment. It is a figure which demonstrates concretely the image | video which the AR viewer A in Embodiment 3 sees through HMD. 12 is a flowchart illustrating a process flow in an AR projection process according to the third embodiment.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Head-mounted display (HMD)>
FIG. 1 is a block diagram showing a hardware configuration of the HMD 100 used in the present embodiment. The HMD 100 includes a transmissive display 101, a camera 102, and a controller 120. The display 101 has a display screen, and displays the real space, that is, the real world and digital image information on the display screen, thereby realizing an augmented reality space on the display screen.

  FIG. 2 is a diagram illustrating a state in which the user 200 wears the HMD 100 having the above configuration. The HMD 100 is similar to a commercially available one, and is attached to the user's 200 head by a frame 130. The transmissive display 101 may be viewed with one eye or viewed with both eyes. The camera 102 included in the HMD 100 can capture images from a viewpoint close to the eyes of the user 200. The user can simultaneously view the digital image information displayed on the display screen of the transmissive display 101 and the real image ahead of the transmissive display 101.

  Returning to FIG. 1, the configuration of the controller 120 will be described in detail.

  The internal bus 112 transmits and receives electrical signals so that information can be transmitted between the memory controller 107 and the I / O controller 109. The memory controller 107 controls all memory accesses to the main memory 108. The I / O controller 109 transmits / receives information to / from each component connected via the ROM 110, the LAN port 103, the transmissive display 101, the MEMS sensor 111, and the internal bus 112. The LAN port 103 transmits / receives information to / from other devices such as the processing server 300 and the I / O controller 109 connected via the wireless access point 400 and / or the network 900 shown in FIG.

  The ROM 110 stores a boot loader program and an HMD control program. The main memory 108 is a volatile memory and can be accessed at high speed, and stores information stored in the ROM 110 and information used temporarily. When the HMD 100 is powered on, the CPU 104 first reads and executes the boot loader program, and then extracts the HMD control program stored in the ROM 110 and stores it in the main memory 108. Then, the CPU 104 executes the HMD control program stored in the main memory 108, and executes each function of the HMD 100. In addition, when the CPU 104 executes this HMD control program, processing performed by the HMD 100 such as data communication with the processing server 300 and information display on the transmissive display 101 based on data acquired by the data communication is executed. . The controller 120 can include other storage devices.

  The transmissive display 101 for displaying information is controlled by a graphic controller 106 included in the controller 120. The HMD 100 displays information on the transmissive display 101 when presenting information to the user. The camera 102 converts the captured image into digital data and outputs it. The camera 102 transmits / receives information to / from the I / O controller 109. The MEMS sensor 111 included in the controller 120 can measure the east, west, north, and south directions, the inclination in the three-dimensional direction, and acceleration, and converts the measured information into digital information. The MEMS sensor 111 transmits / receives information to / from the I / O controller 109. With the MEMS sensor 111, it is possible to grasp in which direction the user wearing the HMD 100 is facing in the physical space. A battery 105 provided in the controller 120 supplies power to the entire HMD 100.

<Processing server>
FIG. 3 is a block diagram showing a hardware configuration of a processing server (information processing server) used in the present embodiment. The processing server 300 includes a controller 320, a display 301, a keyboard 302, and a mouse 303.

  The configuration of the controller 320 will be described in detail.

  The internal bus 311 transmits and receives electrical signals so that information can be transmitted between the memory controller 307 and the I / O controller 309.

  The memory controller 307 controls overall memory access to the main memory 308. The I / O controller 309 transmits / receives information to / from each component connected via the HDD 310, the LAN port 304, the keyboard 302, the mouse 303, and the internal bus 311. The LAN port 304 transmits / receives information to / from the I / O controller 309 and other devices connected via the network 900 or the wireless access point 400 shown in FIG.

  The HDD 310 stores a boot loader program and a processing server control program. The main memory 308 is a volatile memory and can be accessed at high speed, and therefore stores information stored in the HDD 310 and information used temporarily. When the processing server 300 is powered on, the CPU 305 first reads and executes the boot loader program, and then extracts the processing server control program stored in the HDD 310 and stores it in the main memory 308. Then, the CPU 305 executes a processing server control program stored in the main memory 308 and executes each function of the processing server. In addition, when the CPU 305 executes this processing server control program, processing performed by the processing server, such as extraction of an AR content and a person who interrupts between the viewers of the AR content, is executed. Data obtained by the processing is sent to the HMD 100 via the network 900. A display 301 for displaying information is controlled by a graphic controller 306 provided in the controller 320.

  The processing server 300 displays information on the display 301 when presenting information to the user, and the user can input information to the processing server 300 by operating the keyboard 302 or the mouse 303.

<System configuration>
FIG. 4 is a diagram showing a system configuration of the present embodiment. The system of this embodiment includes a plurality of HMDs (Head Mounted Displays) 100 worn by a person, and includes a wireless access point 400, a processing server 300, and a network 900. Each HMD 100 and the wireless access point 400 are connected by radio, and the processing server 300 and the wireless access point 400 are connected by a network 900.

  In this system, images captured by the camera 102 built in the HMD 100 worn by each user and azimuth, acceleration, and tilt information from the MEMS sensor are transmitted to the processing server 300 via the wireless access point 400 and the network 900 one by one. Is done. Based on these pieces of information, the processing server 300 estimates each user's location in the real space. There are existing techniques represented by SLAM (Simultaneously Localization and Mapping) for position estimation techniques from captured images of cameras. In addition, regarding a position estimation technique from a MEMS sensor, there is an existing technique represented by PDR (Pedestrian Dead Reckoning). By using these technologies for the processing of the processing server 300, which HMD 100 user the person captured by the camera built into each user's HMD 100 is determined based on the location information of each user and the azimuth information of the MEMS sensor. Can be specified. If the orientation of the camera 102 incorporated in the HMD 100 and the orientation of the transmissive display 101 are fixedly determined, it is possible to specify which HMD 100 user is in which orientation in the field of view of a certain HMD 100 user.

  In the following description, AR content refers to various digital image information that is a virtual object created by a computer for display on a display. Examples of AR content include a two-dimensional or three-dimensional CG image, a document created by a document creation application, and a PDF file. The AR viewer or browsing user refers to a user of the HMD 100 who has a right to browse a certain AR content, is permitted to browse the AR content on the HMD 100, and is set to be viewable. The AR non-browser or non-browsing user refers to a user of the HMD 100 who does not have the right to browse certain AR content, is prohibited from browsing the AR content on the HMD 100, and is not set to be viewable. The reference AR viewer is a user of the HMD 100 who can see the AR content, the AR viewer, and the non-AR viewer shown in the figure.

(Embodiment 1)
In the present embodiment, the HMD 100 corresponds to a situation in which there is an AR viewer B and an AR non-browser (non-browsing user) C between the AR content and the AR viewer (browsing user) A referring to the AR content. The display on the display screen of the transmissive display 101 included in is controlled. That is, the reference AR viewer A is provided with a control in which the AR viewer B is seen in front of the AR content, and the AR non-viewer C is hidden by the AR content and cannot be seen.

  FIG. 5 is a plan view showing an example of the positional relationship between a person and AR content in the present embodiment.

The AR content 50 has a known display size set in advance, and specifically, the width of the AR content 50 is wt. The display size can be preset by the user, for example, set to a size corresponding to a 60-inch display, set to a size corresponding to an 80-inch display, and held. In the AR content 50, the center of the projection position (AR content display position origin O) is determined by the position information value managed in the AR content position management table (indicated by reference numeral 100 shown in FIG. 10), and the AR content position management table. The direction of projection is determined by the direction information managed by. The distance from the reference AR viewer A to the display position origin O is dist _org . The display size may be held in the storage device of the HMD 100 worn by the user who has made the setting, but the display size may be managed by an AR content position management table (FIG. 10) described later.

  In order to simplify the description, the coordinate space is considered such that the position of the reference AR viewer A is (0, 0) and the display position origin O is in the direction of angle 0 with respect to the horizontal direction from the reference AR viewer A. .

An angle formed by a line connecting the reference AR viewer A and the display position origin O and a line passing through the reference AR viewer A and the non-AR viewer C is α. If the AR non-browser C is at the coordinates (x _c , y _c ), α is as follows:

  If an angle formed by a line connecting the reference AR viewer A and the display position origin O and a line connecting the reference AR viewer A and the upper end of the AR content in the figure is β, β is as follows.

The distance dist _AC between the reference AR viewer A and the non-AR viewer C is expressed by the following equation.

The distance dist _cont between the reference AR viewer A and the AR content in the angle α direction is expressed by the following equation.

  The following two conditions:

If both are satisfied, it can be determined that the AR non-browser C is interrupted between the reference AR viewer A and the AR content 50.

If the values of the coordinates (x _c , y _c ), the display width wt, and the distance dist _org between the reference AR viewer A and the AR content display position origin O are known from the above description, the AR content and the reference AR browsing are known. It can be determined that another AR viewer or AR non-browser has interrupted the user.

  FIG. 6 is a diagram for specifically explaining an image viewed by the reference AR viewer A through the HMD 100 in the present embodiment.

  According to the conventional technique for displaying a person in front of the AR content, not only the AR viewer B but also the AR non-viewer C can be seen in front as shown in FIG. As a result, there has been a problem that an unexpected person can prevent the reference AR viewer A from browsing the AR content 50. In order to solve this problem, in this embodiment, as shown in (b) in the figure, the AR viewer B appears to be in front of the AR content, and the AR non-viewer C appears to be hidden behind the AR content. To display. However, in this case, for example, when the reference AR viewer A points to the AR content 50 by hand without noticing the AR non-browser C that has interrupted the AR content 50, the reference AR viewer A A hand may hit AR non-browser C. Therefore, in the improved form, it is more preferable to add a human contour line 60 as an AR display to the contour of the region where the AR non-browser C exists as shown in FIG. Thereby, in addition to the AR non-browser C not being prevented from browsing the AR content 50, the reference AR viewer A can notice the presence of the AR non-browser C. It is possible to prevent hitting the non-browser C. The display for indicating the presence of the AR non-browser C is not limited to the contour line. For example, the brightness of the AR content 50 in the existing area is different from that in the other areas, the shade is highlighted, and the presence is present. It is conceivable to give luminance flicker to the area to be performed.

  FIG. 7 is a flowchart showing a process flow in the user location registration process according to the present embodiment. By this processing, the information processing system calculates the user position in the real space where the HMD 100 is worn and used based on the information sensed in the real world (real space), and the processing server 300 holds it. This information processing system periodically executes this processing at a predetermined cycle in order to always keep the latest information. The CPU 104 of each HMD 100 shown in FIG. 4 executes the control program stored in the ROM 110, whereby each processing step S701 to S702 of this flowchart is performed. When the CPU 305 of the processing server 300 executes the control program stored in the HDD 310, the processing steps S703 to S704 of this flowchart are performed.

  In step S <b> 701, the HMD 100 performs self-position estimation processing by using a captured image captured by the built-in camera 102 or analyzing a sensor signal output from the MEMS sensor 111. The self-position estimated here is a position where the HMD 100 is present in the real world, and is position information of the user wearing the HMD 100.

  Self-position estimation using captured images includes, for example, a method for estimating a self-position from a landmark as in Non-Patent Document 1, and a method for estimating a self-position using a marker. This document is an example of using natural feature points for landmarks. For example, the MR (Mixed Reality) system (registered trademark) of Canon (registered trademark) is an example in which a marker affixed indoors is used as a landmark.

  Self-position estimation using a MEMS sensor signal is a method that can improve robustness, and is preferably used when self-position estimation using a captured image cannot be performed. A method called PDR (pedestrian dead reckoning) There is. For example, a self-position estimation method using acceleration integration as described in Non-Patent Document 2 can be used.

  Apart from this, it is also possible to adopt a method of estimating the final self-position by correcting the error caused by self-position estimation (PDR) from the MEMS sensor signal with the self-position estimation result from the captured image of the camera. . As this type of method, for example, there is a method as described in Non-Patent Document 3.

  When the self-position estimation is performed with the configuration of Non-Patent Document 3, the direction in which the HMD 100 is facing can be further calculated with the configuration in S701. In this case, it can be replaced with the processing of S1108 shown in FIG.

  When the self-position is estimated in S701, the process proceeds to S702, where the HMD 100 transmits the HMD user information and the estimated user position information to the processing server 300, and the process on the HMD side is terminated.

  In S703, the processing server 300 receives the HMD user information and the user position information from each HMD 100, and proceeds to S704.

  In S704, the processing server 300 updates the position information of the personal attribute information management table (indicated by reference numeral 80 in FIG. 8) in association with the user of each HMD 100, and ends this processing.

  FIG. 8 is a diagram showing an example of the personal attribute information management table held on the HDD 310 by the processing server 300 in this embodiment.

  According to the personal attribute information management table 80, position information that indicates where a person (HMD user) is in the real world as a coordinate value and AR viewer information that indicates which AR content the user has the right to browse. Manage user names as keys. Any known position in the real space, for example, the position of the corner of the living room or presentation venue where the HMD user exists can be defined as the origin of the position information. As coordinate axes in the real space, for example, the north direction is defined as the x-axis direction, the east direction is defined as the y-axis direction, and the vertical direction is defined as the z-axis direction.

  From the personal attribute information management table 80, for example, it can be seen that the user A is at the coordinates (10, 10, 10) and is permitted to browse the content X and the content Z. The AR viewer information can be set for each AR content in advance by a user or system administrator from an information terminal (not shown) such as a PC. The system performs processing (not shown) for reflecting the setting value in the personal attribute information management table 80. Such setting is described in detail in Japanese Patent Application No. 2013-208150 by the present applicant.

  FIG. 9 is a flowchart showing a process flow in the AR projection position determination process according to the present embodiment. When the CPU 104 of each HMD 100 executes the control program stored in the ROM 110, the processing from S901 to S902 of this flowchart is performed. When the CPU 305 executes the control program stored in the HDD 310, the processing from S903 to S904 in this flowchart is performed.

  With this processing, when the AR content projection instruction from the user is given, the information processing system determines a position separated by a predetermined distance from the user as the AR content display position, and the processing server 300 performs the AR content position management table (FIG. 10).

  In step S901, the HMD 100 receives a content projection instruction from the user. Regarding the projection instruction from the user, for example, it may be recognized by analyzing an image captured by the built-in camera 102 including the user's gesture (hand movement). More specifically, when a user frequently puts a finger sack of a different color on the finger and registers the user in association with each color, the user who made a gesture for projection instruction can be identified ( Non-patent documents 4 and 5). Separately, when the camera 102 is configured in a stereo configuration or configured to project infrared rays, depth information in the captured image can be obtained, and a gesture at a predetermined depth is identified based on the depth information and a projection instruction is given. Can be discriminated (see Non-Patent Document 6).

  When the content projection instruction is received from the user, the process proceeds to S902, where the HMD 100 transmits the HMD user information and the AR projection position determination processing instruction to the processing server 300, and ends the processing on the HMD side. The AR projection position determination processing instruction is a command for causing the processing server 300 to perform the process shown in S904 for determining the projection position of the AR content.

  In S903, the processing server 300 receives the HMD user information and the AR projection position determination processing instruction from each HMD 100, and the process proceeds to S904.

  In S904, the processing server 300 determines a location that is a predetermined distance away from the location where the HMD user is present as the AR content display location, updates the location information in the AR content location management table (FIG. 10), and ends this processing. . The AR content display position is calculated by reading the position information of the user who issued the content projection instruction from the personal attribute information management table 80 and adding a predetermined distance set in advance to the position information.

  FIG. 10 is a diagram showing an example of the AR content location management table held in the HDD 310 by the processing server 300 in the present embodiment.

  The AR content position management table 100 uses the content ID of the AR content as a key, and the position information and the display direction information indicating in what position and in what direction the AR content is displayed in the augmented reality space as a coordinate value. Manage. The position information is periodically updated according to the processing result of S904. Any known position in the real space, for example, the position of the corner of the living room or presentation venue where the HMD user exists can be defined as the origin of the position information. As coordinate axes in the real space, for example, the north direction is defined as the x-axis direction, the east direction is defined as the y-axis direction, and the vertical direction is defined as the z-axis direction. The orientation information is expressed as an angle increasing counterclockwise in the xy plane with reference to the x-axis direction.

  In FIG. 10, for example, it is understood that the AR content X is displayed at the coordinates (16, 10, 0), and the display direction of the AR content is the 90 degree direction.

  FIG. 11 is a flowchart showing a process flow in the AR projection process according to the present embodiment.

  It is necessary to change the display position of the AR content display screen according to the movement of the HMD user. By executing this process periodically at a predetermined cycle and controlling the display of the AR content on the display screen of the transmissive display 101, the AR content can be always projected at the correct position on the screen. When the CPU 305 executes the control program stored in the HDD 310, the processing from S1101 to S1105 in this flowchart is performed. When the CPU 104 of each HMD 100 executes the control program stored in the ROM 110, the processing from S1106 to S1112 of this flowchart is performed.

  In S1101, the processing server 300 reads the user location information and AR viewer information from the personal attribute information management table 80, and the process proceeds to S1102.

  In S1102, the processing server 300 reads the display position of the AR content from the AR content position management table 100, and proceeds to S1103.

  In S1103, the processing server 300 determines and extracts a person who has interrupted between the reference AR viewer and the AR content being referred to based on various information. The method for determining whether or not the user has interrupted is as described in the description of FIG. The various types of information include user location information managed by the personal attribute information management table 80, AR content location information managed by the AR content location management table 100, and AR display size information set for the AR content. use.

  In step S1104, the processing server 300 selects the interrupted AR viewer and the interrupted AR non-viewer based on the interrupted user information calculated in S1103 and the AR viewer information managed in the personal attribute information management table 80. Determine, extract, and proceed to S1105.

  In S <b> 1105, the processing server 300 transmits, to the HMD 100 worn by each user, the position information of the interrupted AR viewer and the interrupted AR non-viewer, the display position of the AR content, and the display size information to each HMD 100. And the processing on the processing server side is terminated.

  Each HMD 100 that is the transmission partner of these information executes S1106. In S1106, the HMD 100 receives from the processing server 300 the position information of the interrupted AR viewer and the interrupted AR non-viewer, the AR content display position, and the display size information, and proceeds to S1107.

  In S1107, the HMD 100 uses the HMD 100 of the interrupted AR viewer and the interrupted AR non-viewer based on the stored self-location information and the location information of the interrupted AR viewer and AR non-viewer. The direction with respect to the reference AR viewer himself is calculated.

  In step S1108, the HMD 100 calculates the direction in which the HMD 100 faces by analyzing the position marker and the MEMS sensor signal captured by the built-in camera 102, and the process advances to step S1109. As described above, the direction calculation process can be performed by the method of Non-Patent Document 3, for example.

  In S <b> 1109, the HMD 100 calculates the direction of each human body relative to the reference AR viewer himself who is using the HMD 100. In this process, an image of a human body included in an image captured by the camera 102 is detected, and each human body for the reference AR viewer himself / herself is determined from the orientation information of the HMD 100 calculated in S1108 and the position information of the human body image in the captured image. This is done by calculating the direction of. As a method for detecting a human body image in a captured image, a method widely used in a network camera or the like can be used.

  In S1110, the HMD 100 calculates whether each human body is an AR viewer or an AR non-browser based on various types of information, and proceeds to S1111. As various types of information, the information about the direction of the interrupted AR viewer and the interrupted AR non-viewer with respect to the AR viewer calculated in S1107 and the information on the direction of each human body with respect to the reference AR viewer itself calculated in S1109 are used. To do.

  In S1111, the HMD 100, based on the orientation information of the HMD 100, the location information held by the reference AR viewer who uses the HMD 100, and the display location information, size information, and orientation information of the AR content, The AR content to be projected is generated. Each HMD 100 holds data for AR content in a storage device. The data for AR content may be held by the processing server 300 and provided to the HMD 100 that has issued the projection instruction from the processing server 300 in response to the projection instruction.

  The process performed in S1111 is a process of converting the coordinate space on the display screen of the transmissive display 101 of the HMD 100 using the coordinate conversion matrix, which is the above-described coordinate space defined as an origin in the real world. The coordinate transformation matrix is expressed by the following equation.

  For this conversion, various libraries such as ARTToolKit are released.

  In S1112, the HMD 100 performs a display display process shown in another figure and ends this process. The processing of S1112 will be described next with reference to FIG.

  FIG. 12 is a flowchart showing a process flow in the display display process executed in S1112 of FIG. 11 in the present embodiment. When the CPU 104 of each HMD 100 executes a control program stored in the ROM 110, the processing from S1201 to S1203 of this flowchart is performed.

  In step S1201, the HMD 100 deletes the AR content in an area that overlaps the area where the human body corresponding to the interrupted AR viewer exists, and the process advances to step S1202. The detection of the region where the human body exists will be described later.

  In step S1202, the HMD 100 adds a human outline to the AR content regarding the area where the human body corresponding to the interrupted AR non-viewer exists, and the process advances to step S1203.

  In step S1203, the HMD 100 displays the AR content on the display screen of the transmissive display 101, and ends this process.

  In S1109 of FIG. 11, the HMD 100 calculates the direction of the human body included in the video captured by the camera 102 for the reference AR viewer, and will be supplementarily described with reference to FIGS. 13 and 14.

  FIG. 13 is a diagram showing the field of view of the AR viewer in the present embodiment.

  When the human body 1300 included in the video imaged by the camera 102 exists in the direction of the angle γ with respect to the normal direction 1310 of the camera, the reference AR viewer also has the same angle with respect to the normal direction of the transmissive display 101 of the HMD 100. The human body 1300 is seen in the direction of γ. Consider a case where there is a region where the human body 1300 overlaps AR content (not shown) in the horizontal direction. At this time, the display in which the human body 1300 appears to be positioned in front of the AR content can be expressed by the HMD 100 displaying the AR content by deleting the AR content on the display screen 1320 for the region H in the horizontal direction. On the other hand, the display in which the human body 1300 appears to be located behind the AR content can be expressed by displaying the AR content on the display screen 1320 for the region H as well.

  FIG. 14 is a diagram showing the direction of the human body included in the video captured by the camera 102 with respect to the reference AR viewer as viewed from above in the present embodiment. If the direction to which the reference AR viewer is facing is the angle θ, the other person will see the direction of the angle θ + γ as seen from the reference AR viewer. The angle θ is obtained by analyzing a marker photographed by the camera, or by calculating a relative direction with respect to the direction that was previously directed from the MEMS sensor signal.

The user's positional relationship can be grasped from the personal attribute information management table 80. For example, if the reference AR viewer is located at the coordinates (x _o , x _o ) and the other is located at the coordinates (x _t , y _t ), the angle at which the other person can see the reference AR viewer is It can be expressed in terms of an expression.

  Regarding S1201, a supplementary explanation will be given regarding a method of calculating a human body region in a camera image and drawing a contour line of the human body region. For example, there are a method of calculating from a depth image, a method of calculating by recognizing a human shape, and the like. In order to acquire a depth image, for example, a method using infrared rays adopted by Kinect (registered trademark) of Microsoft (registered trademark) or RGB as in DTAM (Dense Tracking and Mapping) of non-patent document 7 There is a method of calculating from an image.

  Assuming that the reference AR viewer is located at the coordinates (xo, xo) and the other person is located at the coordinates (xt, yt), the distance between them is as follows.

  As described above, the direction in which a person can see in the camera image is known. A contour line can be drawn by searching depth information that is not greatly deviated from the direction in which a person can see and the distance between the two and drawing a line on the outer periphery thereof.

  With the above processing, in a situation where there is an AR viewer B and an AR non-browser C between the AR content and the reference AR viewer A, the AR viewer B appears to the reference AR viewer A in front of the AR content, and the AR The non-browser C is controlled so as not to be hidden by the AR content. Here, since the process of FIG. 11 is periodically performed at a predetermined cycle, when the AR non-browser C acquires the AR content viewing right in that state, the user C who becomes the AR viewer refers to the reference AR. As seen from the viewer A, it is possible to realize control in which the appearance changes dynamically.

(Embodiment 2)
In the present embodiment, the AR viewer who is deeply related to the AR content among the interrupted AR viewers and is supposed to be noticed by the reference AR viewer is displayed in front of the AR content.

  In the first embodiment, there is an AR viewer B and an AR non-browser C explaining the AR content between the AR content and the reference AR viewer A, and the reference AR viewer A is an AR viewer B. Is seen in front of the AR content, and the AR non-browser C is hidden by the AR content and cannot be seen.

  In the present embodiment, another situation is assumed in which there is not only the AR viewer B explaining the AR content but also the AR viewer E who is only one of the audience between the reference AR viewer A and the AR content. . In this situation, the AR viewer B is deeply related to the AR content and is preferably visible to the reference AR viewer A, and the AR viewer E does not have to be particularly visible, but rather the AR content is visible. Browsing may be hindered. Therefore, in the present embodiment, the display on the display screen is controlled so that the AR viewer B can be seen by the reference AR viewer A and the AR viewer E can not be seen even by the AR viewer.

  FIG. 15 is a plan view showing an example of the positional relationship between the user and the AR content.

  Users A, B, and E in the figure are all AR viewers. The user B stands next to the AR content 15 and explains the AR content 15, and the user A and the user E are illustrated assuming that they are watching it. In this case, for the AR viewer A, the AR viewer B is a notable user, that is, a user closely related to the AR content 15. In the present embodiment, a user who is present in front of the AR content 15 as viewed from the reference AR viewer A and stands within a predetermined distance (for example, within 1 m) from the AR content 15 is regarded as a person to be noted. In addition, whether or not to pay attention is determined based on whether or not the user is speaking (whether the sound detected by the microphone is equal to or higher than a predetermined level), or whether the body movement is large (the fluctuation of the MEMS sensor signal is large). You can also.

In order to simplify the explanation, consider a coordinate space where the position of the reference AR viewer A is (0, 0) and the display position origin O is in the direction of angle 0 from the reference AR viewer A. Assume that the AR viewer B is at coordinates (x _B , y _B ), and the coordinates of the display position origin O of the AR content 15 are (x _cont , 0). The length of the line segment perpendicular to the AR content 15 from the AR viewer B, that is, the distance dist between the AR viewer B and the AR content 15 is expressed by the following equation.

  In this embodiment, the following two conditions:

When both are satisfied, the AR viewer B can be regarded as a notable person.

  FIG. 16 is a diagram for specifically explaining an image viewed by the reference AR viewer A through the HMD 100 in the present embodiment.

  In the display control of the first embodiment described for comparison, as shown in FIG. 16A, not only the AR viewer B who is the presenter but also the AR viewer E who is the mere audience will be seen in the foreground. . Thereby, the reference AR viewer A may be prevented from browsing the AR content 15 by an audience who does not need to pay attention. Therefore, in the present embodiment, as shown in FIG. 16B, the AR viewer B who is a notable explainer is displayed so as to be positioned in front of the AR content 15, and the AR viewer E is displayed as the AR content 15. Display hidden from view. Thereby, the reference AR viewer A can provide more improved control that is not hindered from browsing the AR content 15.

  The control of the present embodiment can be realized by the processing shown in FIGS. 7, 9, and 11. However, the processing executed in S1112 of FIG. 11 can be realized by replacing the processing of FIG. 12 with the processing of FIG. .

  FIG. 17 is a flowchart showing a process flow in the display display process executed in S1112 of FIG. 11 in the present embodiment. When the CPU 104 of each HMD 100 executes the control program stored in the ROM 110, the processing from S1701 to S1703 in this flowchart is performed.

  In S <b> 1701, the HMD 100 determines whether the AR viewer is within a distance of 1 m from the AR content 15 according to the description of FIG. 15, and the AR overlaps with the region where the human body corresponding to the AR viewer B within 1 m exists. The content is deleted, and the process proceeds to S1702.

  In S <b> 1702, the HMD 100 adds an image representing a human-type contour line to the AR content regarding an area where human bodies corresponding to other AR viewers and AR non-browser exist, and the process proceeds to S <b> 1703. The image representing the human outline can be generated by a method similar to the method described in S1202.

  In step S <b> 1703, the HMD 100 displays the AR content including the contour line on the display screen of the transmissive display 101 in the AR content in which the region overlapping the human body existing region is erased, and the process ends.

(Embodiment 3)
In the first and second embodiments, the AR content in which the area overlapping the display in the augmented reality space of the AR viewer interrupted by the system control is deleted is displayed on the display screen of the transparent display, and the interrupted AR viewer displays the AR content. It was controlled so that it was visible in the foreground. However, since it may not be desired to display a person in front of the AR content, this embodiment corresponds to such a case, and the interrupted AR viewer can be seen in front of the AR content in accordance with a user instruction. It is comprised so that control which displays may not be performed.

  FIG. 18 is a diagram for specifically explaining an image viewed by the reference AR viewer A through the HMD 100 in the present embodiment. In this embodiment, as shown in the figure, no person, that is, a user is displayed in front of the AR content.

  The processing of this embodiment follows the processing shown in FIGS. 7, 9, and 11, but differs in that the AR projection processing of FIG. 19 is executed instead of S 1106 to S 112 of FIG. 11.

  FIG. 19 is a flowchart showing the flow of processing in the AR projection processing according to this embodiment. When the CPU 104 provided in each HMD 100 executes the control program stored in the ROM 110, the processing from S1901 to S1905 of this flowchart is performed.

  In S1901, the HMD 100 receives, from the processing server 300, the location information of the interrupted AR viewer and the interrupted AR non-viewer, the AR content display position, and the display size information, and proceeds to S1902.

  In S1902, the HMD 100 receives a person non-display instruction from the user, and proceeds to S1903. As for the non-display instruction of the person from the user, a method of recognizing the user's gesture as mentioned in the description of S901 can be used.

  In S1903, the HMD 100 analyzes the position marker photographed by the built-in camera 102, calculates the direction in which the HMD 100 is facing, and proceeds to S1904. This processing can be performed in the same manner as the processing mentioned above in S1108.

  In step S1904, the HMD 100 generates an AR content to be projected based on the orientation information of the HMD 100, the AR content display position information, and the AR content size information, and the process advances to step S1905.

In step S1905, the HMD 100 displays the AR content on the display screen of the transmissive display 101, and ends this process. This process can be performed in the same manner as the process mentioned above in S1203.

  In addition, although each said embodiment applies this invention to HMD provided with a transmissive display, this invention can also be applied also to HMD provided with a non-transmissive display. Furthermore, the present invention can also be applied to an information processing system including the HMD, an information processing server provided in the information processing system, an information processing method in the information processing system, a program used in the HMD and the information processing server. it can. In this case, the image of the interrupted AR viewer is extracted from the captured image captured by the camera provided in the HMD, the AR content is erased in a region where the image overlaps the display of the AR content, and the AR content is displayed as the captured image. Control to superimpose on the mask. By such control, even on the display screen of the non-transparent display, it is possible to display the reference AR viewer so that the interrupted AR viewer appears to be positioned in front of the AR content. On the other hand, the control for preventing the interrupted AR viewer from being hidden behind the AR content and the control for drawing the outline in the area where the AR viewer is present are performed by using the non-transparent display. It is realizable similarly to the case of embodiment.

(Other embodiments)
The present invention can be applied to a system constituted by a plurality of devices (for example, a computer, an interface device, a reader, a printer, etc.), or can be applied to an apparatus (multifunction device, printer, facsimile machine, etc.) comprising a single device. Is also possible.

  A processing method for storing the program for operating the configuration of the above-described embodiment so as to realize the function of the above-described embodiment in a storage medium, reading the program stored in the storage medium as a code, and executing the program on the computer is also described above. It is included in the category of the embodiment. That is, a computer-readable storage medium is also included in the scope of the embodiments. In addition to the storage medium storing the computer program, the computer program itself is included in the above-described embodiment.

  As such a storage medium, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, and a ROM can be used.

  In addition, the processing is not limited to the single program stored in the above-described storage medium, but operates on the OS in cooperation with other software and expansion board functions to execute the operations of the above-described embodiments. This is also included in the category of the embodiment described above.

Claims (10)

In an information processing system comprising a display device that displays real space and digital image information on a display screen and presents the augmented reality space on the display screen,
First acquisition means for acquiring first information indicating the position of the display device in the real space;
Second acquisition means for acquiring second information indicating whether or not a user wearing the display device is set to be able to view the digital image information;
Third acquisition means for acquiring third information indicating a position and a size for displaying the digital image information in the augmented reality space;
Based on the first to third information, for each user, the digital image information that exists between the position in the real space corresponding to the position for display and the existing position is set to be viewable. Determining means for determining a browsing user and a non-browsing user not set, and
By displaying the digital image information and the real space in which the image of the region overlapping the display in the augmented reality space of the browsing user is displayed on the display screen, the browsing user in the augmented reality space from the digital image information Control means for controlling display on the display screen so that the non-browsing user is positioned in front of the digital image information;
An information processing system comprising: The control means includes
For the viewing user who is highly related to the digital image information, the digital image information is erased and displayed on the display screen in an area overlapping with the viewing user's display in the augmented reality space, and the digital image For the viewing user who is not highly related to information, the display on the display screen is controlled so that the digital image information is displayed on the display screen without erasing the image of the area overlapping the viewing user. The information processing system according to claim 1.   The control means includes that the distance between the browsing user and the digital image information in the augmented reality space is within a predetermined distance, that the browsing user is speaking, and that the browsing user moves a lot. The information processing system according to claim 2, further comprising means for determining that the relevance is high when at least one is detected. The control means further includes
Display on the display screen such that information indicating that there is an area overlapping with the non-viewing user is added to the digital image information and the real space and the digital image information are displayed on the display screen. The information processing system according to claim 1, wherein the information processing system is controlled.   The information processing system according to claim 3, wherein the information is an image of a contour of an area where the digital image information overlaps with the non-browsing user.   The control by the control means for displaying on the display screen the digital image information obtained by erasing an image of an area overlapping the display in the augmented reality space of the browsing user is not performed by an instruction from the user. Item 6. The information processing system according to any one of Items 1 to 5. In an information processing apparatus comprising a display device that displays real space and digital image information on a display screen and presents the augmented reality space on the display screen,
Means for obtaining information indicating the position of the display device in the real space;
Means for sensing the real space and obtaining information about an object existing in the real space;
A viewing user set to be able to view the digital image information and a non-viewing user that is not set, the position in the real space corresponding to the position for displaying the digital image information in the augmented reality space, and the Means for acquiring information about a browsing user and a non-browsing user existing between the existing position and associating the browsing user and the non-browsing user with the object based on the information and the information about the object; and
By displaying the digital image information and the real space in which the image of the region overlapping the display in the augmented reality space of the first object in the real space corresponding to the browsing user is displayed on the display screen, The display so that the first object is located in front of the digital image information in the real space and the second object in the real space corresponding to the non-viewing user is located behind the digital image information. Means for controlling the display on the screen,
An information processing apparatus comprising: In an information processing server that provides information to a display device that displays real space and digital image information on a display screen and presents the augmented reality space on the display screen,
First acquisition means for acquiring first information indicating the position of the display device in the real space from the display device;
Second acquisition means for acquiring second information indicating whether or not a user wearing the display device is set to be able to view the digital image information;
Third acquisition means for acquiring third information indicating a position and a size for displaying the digital image information in the augmented reality space;
Based on the first to third information, for each user, the digital image information that exists between the position in the real space corresponding to the position for display and the existing position is set to be viewable. Determining means for determining a browsing user and a non-browsing user not set, and
Means for providing the display device with the first information and the third information about the browsing user and the non-viewing user;
An information processing server comprising: In an information processing method using a display device that displays real space and digital image information on a display screen and presents an augmented reality space on the display screen,
First information indicating the location of the display device in the real space, second information indicating whether or not a user wearing the display device is set to view the digital image information, and the extension Obtaining third information indicating the position and size for display of the digital image information in real space;
Based on the first to third information, for each user, the digital image information that exists between the position in the real space corresponding to the position for display and the existing position is set to be viewable. Determining a browsing user and a non-browsing user not set, and
By displaying the digital image information and the real space in which the image of the region overlapping the display in the augmented reality space of the browsing user is displayed on the display screen, the browsing user in the augmented reality space from the digital image information Controlling the display on the display screen so that the non-browsing user is located in front of the digital image information;
An information processing method comprising:   The program for functioning a computer as each means of Claim 7 or 8.