JP7261121B2 - Information terminal device and program - Google Patents

Description

The present invention relates to an information terminal device, a program, a presentation system, and a server that can simply share information from other terminals.

Augmented reality that superimposes presentation information on imaging information is widely known. However, there is a problem that the usable range is limited when shared by a plurality of terminals, and various countermeasures have been taken in conventional augmented reality.

For example, in Japanese Unexamined Patent Application Publication No. 2002-100000, a server creates a panoramic composite image of a stadium from a plurality of wide cameras and recognizes the positions of athletes and the like. The recognized position in the panoramic composite image is distributed to multiple terminals, and each terminal recognizes the position and posture of the terminal based on the field line of the stadium, and superimposes related information according to the recognition result.

In Non-Patent Document 1, in order to allow the vehicle behind to see the scene blocked by the vehicle ahead, the vehicle ahead transmits image information to the vehicle behind, and the image information of the front and the image information of the vehicle behind correspond to feature points. is applied to the imaging information of the forward vehicle, and synthesized with the imaging information of the rear vehicle.

In Non-Patent Document 2, in order to allow the rear vehicle to confirm the scene blocked by the front vehicle, the front vehicle transmits the three-dimensionally restored point cloud and feature values to the rear vehicle, and the self-position estimation result of the rear vehicle. is applied to the imaging information of the forward vehicle and combined with the imaging information of the rear vehicle.

In Patent Document 2, in short-range communication with other terminals, by receiving position information determined by other terminals and information depending on the surrounding environment of other terminals (such as signal strength of other terminals), the surrounding environment of other terminals can be detected. Information that depends on the position is linked to the position and presented to the imaging information.

Retable 2016-017121 JP 2017-143409 A

"Occluded Street Objects Perception Algorithm of Intelligent Vehicles Based on 3D Projection Model", JAT, 2018. "A Real-time Augmented Reality System to See-Through Cars", IEEE TVCG, 2016.

However, the conventional techniques as described above have the following problems.

Patent document 1 is limited to a space that can be photographed by a wide camera, so there is a problem that it is difficult to widen the area. Also, there is a problem that the server is highly likely to fail in image recognition. For example, players cannot be detected and identified if the players are overlapped and their jersey numbers are blocked. In the first place, there is a problem that related information cannot be superimposed unless there is an object for identifying an individual, such as a jersey number.

Non-Patent Documents 1 and 2 also have the same problem. In other words, even if the type of image, such as a car or a person, is image-recognized from the imaging information, no means is disclosed for disclosing individual information. Moreover, there is a problem that it does not function unless the front vehicle and the rear vehicle capture the same scene. For example, under the situation shown in FIG. 4, the same scene is not imaged in the imaging information of the forward vehicle VB and the imaging information of the rearward vehicle VA, so it does not function at all. (Note that FIG. 4 will be described later for another explanation. Here, FIG. 4 schematically represents that the situation where the function does not work at all occurs due to the following reason. That is, field F1 such as street corner In FIG. 4, the front vehicle VB captures an image in its running direction dB, and the rear vehicle VA captures an image in its running direction dA. ) and the existence of the wall W1 of the building as a shield, the imaged information of the front and rear vehicles do not include a common scene, so a situation in which it does not function at all occurs.)

Furthermore, in Non-Patent Documents 1 and 2, even when the same scene is captured, there is a high possibility that feature point correspondence, three-dimensional reconstruction, and self-position estimation will fail at night or in bad weather, when the same function is expected to be used. There is a problem. In addition, since the field of view is reproduced when there is no shielding object, there is a possibility that the shielding object itself may be lost, and there is a possibility that the original purpose of avoiding danger may not be met.

Patent Document 2 partially solves the above problem. In other words, receiving information from other terminals eliminates the need for an imaging unit that takes a bird's-eye view of the whole, so that an individual can be identified without depending on the weather or the time of day. However, when the number of other terminals is large, the presentation of an image depicting the other terminals, the trajectory, and the like may be troublesome for the viewer. Also, drawing from an arbitrary viewpoint is not displayed.

SUMMARY OF THE INVENTION In view of the problems of the prior art as described above, it is an object of the present invention to provide an information terminal device, a program, a presentation system, and a server that can simply implement information sharing from other terminals.

To achieve the above object, the present invention provides an information terminal device comprising: a measuring unit that measures its own position and orientation; position information measured by one or more other terminals; By referring to, in a three-dimensional virtual space based on the measured position and orientation of itself and a presentation unit for presenting the drawing information. Further, the present invention is characterized by a program that causes a computer to function as the information terminal device. Further, the present invention is a presentation system comprising two or more information terminal devices and a server, wherein each of the information terminal devices includes a measurement unit that measures its own position and orientation, and disclosure information that is set to be public. and a disclosure unit that receives the position and orientation information and the disclosure information to the server, and the server refers to the position and orientation information and the disclosure information received from each of the information terminal devices. , for each of the information terminal devices, in a three-dimensional virtual space based on the position and orientation of the information terminal device, disclosing information of one or more other terminals is arranged at the position of the other terminal and drawn. A drawing unit that obtains drawing information, transmits the drawing information to a corresponding information terminal device, and each of the information terminal devices further includes a presentation unit that presents the transmitted drawing information. . Furthermore, the present invention is characterized by being a server in the presentation system.

According to the present invention, it is possible to simply implement information sharing by setting disclosure information to be actively disclosed on other terminals, arranging the disclosure information in a virtual space, drawing it, and presenting it. can.

It is a figure which shows the structural example of the presentation system which concerns on one Embodiment. 1 is a functional block diagram of an information terminal device and a server of a presentation system according to one embodiment; FIG. FIG. 2 illustrates real-time operation of a presentation system according to one embodiment; FIG. 3 is a schematic diagram showing a street corner field as a model example of the real world in which an imaging unit captures an image, as viewed from above. 5 is a diagram showing a schematic example of a captured image captured in the situation of FIG. 4; FIG. 4 is a functional block diagram of a measurement unit according to one embodiment; FIG. It is a figure which shows the model example of depth information. FIG. 4 is a diagram showing a schematic example of a virtual space drawn by a drawing unit; 9A and 9B are diagrams showing schematic examples of drawing by the drawing unit and superimposed display by the presentation unit in the virtual space of FIG. 8 with respect to various embodiments; FIG. 10 is a diagram showing a schematic example of arrangement of virtual space and the like when performing communication and the like using avatars as an application example of the second embodiment. FIG. 11 is a diagram showing a schematic example corresponding to FIG. 10 for explaining the second embodiment; It is a figure which shows the schematic example of the modification of 2nd embodiment. 1 is a diagram generally showing a hardware configuration in a computer device; FIG.

FIG. 1 is a diagram showing a configuration example of a presentation system according to one embodiment. and a server 20, which are configured to be able to communicate with each other through the network NW. In FIG. 1, three information terminal devices for three users are provided in the presentation system 100, but this is merely an example, and any number of (two or more) information terminal devices may be provided.

In the following description, when referring to the three information terminal devices 10-A, 10-B, and 10-C or a portion thereof, an example for explanation (in particular, one own user A and its own terminal and the information terminal device 10-A as one or more arbitrary number of other users B, C, etc. and the information terminal devices 10-B, 10-C, etc. as one or more other terminals. Each description can be generalized when the presentation system 100 is configured with any plurality of (at least two) information terminal devices.

Each of the plurality of information terminal devices 10-A, 10-B, 10-C, etc. may be configured as a smartphone terminal or the like, and may be carried and moved by users A, B, C, etc., It may move by being configured as a device installed in a moving means such as a vehicle used by users A, B, C, and the like.

FIG. 2 is a functional block diagram of the information terminal device 10 and server 20 of the presentation system 100 according to one embodiment. The information terminal device 10 includes an imaging unit 11, a measurement unit 12, a disclosure unit 13, and a presentation unit 14, and the server 20 includes a storage unit 21 and a drawing unit 22. The information terminal device 10 shown in FIG. 2 is a functional block diagram common to each of the information terminal devices 10-A, 10-B, and 10-C showing three cases as an arbitrary plurality of examples in FIG. That is, each of the information terminal devices 10-A, 10-B, and 10-C in FIG. 1 may have all or part of the configuration of the information terminal device 10 in FIG.

The general operation of the presentation system 100 according to one embodiment is as follows, and can be roughly divided into three processes: (1) information acquisition, (2) rendering, and (3) presentation.

(1) Acquisition of information: imaging unit 11, measurement unit 12, and disclosure unit 13
First, on the information terminal device 10 side, the imaging unit 11 outputs a captured image obtained by performing imaging to the measurement unit 12 and the presentation unit 14 . The measurement unit 12 acquires positioning information including information on the position and orientation of the information terminal device 10 by analyzing the captured image or using positioning means such as GPS (Global Positioning System), and sends the measurement information to the server 20. Send to the storage unit 21 . The disclosure unit 13 receives disclosure information, which is information set to be disclosed to other users as an augmented reality display, by input from a user using the information terminal device 10, and transmits the disclosure information to the storage unit 21 of the server 20. .

(2) Drawing...storage unit 21 and drawing unit 22
On the other hand, on the side of the server 20, the storage unit 21 assigns the user ID (identifier) to the measurement information and disclosure information transmitted from the measurement unit 12 and the disclosure unit 13 of each user's information terminal device 10 as those of the user. By linking and saving, reference from the drawing unit 22 becomes possible. The drawing unit 22 refers to the storage unit 21, draws drawing information corresponding to the information terminal device 10 of each user, and transmits the drawing information to the presentation unit 14 of the information terminal device 10 of the corresponding user.

(3) Presentation: presentation unit 14
In the information terminal device 10, the presentation unit 14 presents the captured image obtained by the imaging unit 11 with the drawing information obtained from the drawing unit 22 superimposed to the user as presentation information realizing an augmented reality display.

For example, when there are three information terminal devices 10-A, 10-B, and 10-C of users A, B, and C in the presentation system 100 as shown in FIG. Drawing information for A's information terminal device 10-A can be drawn as follows. First, since the drawing information is to be superimposed on the image captured by the imaging unit 11 (camera) in the information terminal device 10 of user A, the drawing information is drawn in the camera coordinate system based on the camera's external parameters (representing the position and orientation of the camera). , the drawing unit 22 acquires external parameter information included in the measurement information of the user A from the storage unit 21 .

Then, the drawing unit 22 refers to the measurement information and the disclosure information stored for each of the users B and C, who are other users, from the storage unit 21, thereby obtaining a camera image based on the external parameters (position and orientation) of the camera of the user A. By drawing the disclosure information of the other users B and C in the coordinate system, the drawing information of the user A is obtained as superimposed and presented on the captured image in the presentation unit 14 of the information terminal device 10 of the user A. Details of this will be described later, but by comparing the measurement information of user A with the measurement information of other users B and C, if it is determined that it is outside the range to be displayed as augmented reality, the other user Disclosure information of users B and C may not be drawn, or the manner of drawing may be changed.

Similarly, when the number of information terminal devices 10 present in the presentation system 100 is not the three information terminal devices 10-A, 10-B, and 10-C, but an arbitrary plurality of information terminal devices 10-A, 10-B, and 10-C, Drawing information is obtained by selectively drawing disclosure information of other users in the position and orientation of the own user on the information terminal device 10 of the user ("own user").

In the presentation system 100, the server 20 acquires the measurement information and the disclosure information from the information terminal device 10 of each user in real time, draws the image, and can present the drawing information in real time on the information terminal device 10 of each user. . FIG. 3 shows the real-time operation of the presentation system 100 according to one embodiment as a group of common processing steps S(t) repeatedly executed at each time t (t=1, 2, 3, . . . ), 3 is a diagram showing processing steps of each device in the presentation system 100. FIG. In FIG. 3 , the downward direction represents the direction of time progression, and the horizontal direction represents how information is transmitted and received between each information terminal device 10 and the server 20 .

In FIG. 3, the processing steps of each device are shown at the time t of the aforementioned processing of (1) information acquisition, (2) drawing, and (3) presentation. is as follows. In step GA(t), information is acquired by itself (captured image by the imaging unit 11, measurement information by the measurement unit 12, and disclosure information by the disclosure unit 13). It is transmitted to the storage unit of the server 20 as information associated with the terminal device 10-A). At step RA(t), the drawing unit 22 of the server 20 draws drawing information for the user A (the information terminal device 10-A) and transmits the drawing information to the information terminal device 10-A. At step DA(t), the presentation unit 14 of the information terminal device 10-A presents to the user A an augmented reality display obtained by superimposing the received drawing information on the captured image.

Similarly in FIG. 3, steps GB(t), RB(t), and DB(t) are processing steps for acquiring, drawing, and presenting information at time t regarding the information terminal device 10-B, and step GC(t ), RC(t) and DC(t) are the processing steps for information acquisition, rendering and presentation at time t for the information terminal device 10-C.

As outlined above, the functional units of the presentation system 100 capable of performing real-time augmented reality display by common processing for each information terminal device 10 will be described in detail below.

(imaging unit 11)
The imaging unit 11 captures a scene to obtain a captured image, and outputs the captured image to the measurement unit 12 and the presentation unit 14 . (It should be noted that the output of the captured image to the measuring unit 12 is only required when the measuring unit 12 needs it, and in an embodiment in which the measuring unit 12 does not require the captured image, this output may be omitted.) The imaging unit 11 A digital camera or the like standardly installed in a mobile terminal such as a smart phone can be used as hardware for realizing the above.

FIG. 4 is a schematic diagram showing a street corner field F1 as a model example of the real world in which the imaging unit 11 captures an image, as viewed from above. A wall W1 and information terminal devices 10-A, 10-B, and 10-C of users A, B, and C exist in the field F1. , dB, and the information terminal device 10-C is carried by a user C who is a pedestrian.

FIG. 5 is a diagram showing a schematic example of a captured image captured in the situation of FIG. 4. Specifically, the imaging unit 11 of the information terminal device 10-A in FIG. 4 captures an image in the forward direction dA. A captured image PA obtained by this is shown as a schematic example. In the imaged image PA, the scene dA in front of the vehicle 10-A of the user A is imaged. A vehicle VB in which the user B is riding is imaged, and a wall W1 is imaged on the left side of the image. As shown in FIG. 4, the user C holding the information terminal device 10-C exists behind the wall W1. -C is not imaged. Here, the user C and the information terminal device 10-C are positioned within the range captured in the captured image PA if it is assumed that the wall W1 does not exist and thus no shielding occurs. It should be noted that the examples of FIGS. 4 and 5 will also be referred to in the following explanation examples as appropriate, as they represent the prerequisite conditions.

(Measurement part 12)
The measurement unit 12 acquires measurement information including at least position and orientation information of the information terminal device 10 and transmits the measurement information to the storage unit 21 of the server 20 . When acquiring the measurement information, the captured image obtained by the imaging unit 11 may be used, or a method that does not use the captured image may be used. FIG. 6 is a functional block diagram of the measurement unit 12 according to one embodiment. It includes a depth acquisition unit 123 and a light source information acquisition unit 124 that acquires light source information. Each of these units 121 to 124 may also use a method that uses a captured image or a method that does not use the captured image when acquiring each piece of information.

The measurement unit 12 includes at least the position/orientation acquisition unit 121 to output the measurement information including the position/orientation information. , all or part of the light source information may be output as the measurement information.

The position/orientation acquisition unit 121 acquires external parameters of a camera that configures the imaging unit 11 as hardware as position/orientation information. At this time, any existing method may be used. For example, position and orientation information may be acquired using a GPS, compass (orientation sensor), or gyro (angle sensor) as a positioning sensor that is standardly installed in a mobile terminal such as a smartphone. In addition, by extracting image feature information such as SIFT feature amount from the captured image obtained by the imaging unit 11, a marker or the like whose three-dimensional coordinate information is known is detected in the captured image. The position and orientation information may be acquired from the conversion relationship between the coordinates of the marker in the image and the three-dimensional coordinate information. Alternatively, the position and orientation information may be acquired by triangulation or the like after estimating the distance to the wireless station based on the radio wave intensity or the like from the wireless station using an indoor GPS method. The position and orientation information may be obtained by combining these existing methods.

The internal parameter acquisition unit 122 acquires internal parameters (angle of view and focal length) of a camera that configures the imaging unit 11 as hardware. Intrinsic parameters may be obtained from captured images using the same camera calibration method as in obtaining external parameters in the position/orientation obtaining unit 121, or may be obtained from an actuator that controls the camera that constitutes the imaging unit 11. can be accessed, the internal parameters may be obtained from the drive control signal for this actuator, or the internal parameters are linked to the captured image in the form of Exif information representing the imaging situation by the camera that constitutes the imaging unit 11. may be obtained from this record.

When it is known that the internal parameters do not change over time, the internal parameter acquisition unit 122 acquires the internal parameters only at a certain time and transmits them to the storage unit 21. In the storage unit 21, the Assuming that the internal parameters do not change over time, the rendering unit 22 may continue to refer to them at subsequent times.

Note that the camera parameters (external parameters and internal parameters) are known as mathematical relationships in the field of 3DCG, such as "three-dimensional world coordinate system ⇔ three-dimensional camera coordinate system ⇒ two-dimensional projection coordinate system ⇒ viewport (imaging The image plane of the image)” corresponds to the transformation relationship (viewing pipeline relationship). The extrinsic parameters give the transformation between the former three-dimensional coordinate systems, and the intrinsic parameters give the latter transformation relation from the three-dimensional camera coordinate system to the two-dimensional image plane.

The depth acquisition unit 123 acquires depth information of the captured image (a depth map that gives the depth at which the object being imaged is located at each pixel position of the captured image). Depth information can be acquired using any existing method, and after obtaining the same point correspondence between images captured at two different times (current time and past time close to this), it can be obtained by motion parallax or stereo vision. Alternatively, if a light sensor of a LiDAR (light detection and ranging) device associated with the camera of the imaging unit 11 can be used in the form of in-vehicle equipment, etc., the distance measurement result by the LiDAR device can be used. can be

FIG. 7 is a diagram showing a schematic example of depth information. Specifically, the depth information D(PA) corresponding to the captured image PA in FIG. It is shown in the form of a rendered grayscale image. (The black rectangular frame indicates the image range, not the depth.) The depth information D(PA) in FIG. 7 indicates that the depth of the wall W1 shown in FIG. The depth is intermediate, and the other areas are ground areas such as roads. The area in front of the vehicle VB has an intermediate depth. Consists of greater depth.

The light source information acquisition unit 124 acquires light source information in the surrounding environment where the imaging unit 11 is capturing an image (the surrounding environment where the information terminal device 10 exists). Any existing method may be used to acquire the light source information, such as direct measurement with an illuminance sensor, or an existing method of estimating from the captured image obtained by the imaging unit 11 by deep learning. In addition, if the surrounding environment is composed of indoor lighting, and if this indoor lighting is program-controlled by a lighting control device, etc., access the control device and determine the shape and arrangement of the lighting that is being program-controlled. - The color scheme, illuminance, etc. may be directly captured as light source information. In an outdoor environment, it is calculated from the latitude and longitude information, the imaged direction information, and the date/time information (the date/time information when the captured image was captured) obtained from the position/orientation information by the position/orientation acquisition unit 121. The solar coordinate information and weather information in the captured image may be used as the light source information. The sun coordinate information and weather information may be obtained by accessing a server on the network that holds such information and collating the position and orientation information and the date and time information.

(Disclosure 13)
Disclosure unit 13 transmits disclosure information to storage unit 21 . Depending on the desired application of augmented reality with the presentation system 100, the disclosure can be customized by the user to include any type and content of multimedia content, such as text, images, effects, and the like. The external content may be quoted using a URL or the like (access information to the external content may be given to other users as disclosed information). If the disclosure information is static information that does not change with time (including information that shows a fixed behavior (such as repeating the same action) with respect to the future using time as a parameter), the disclosure unit 13 will only disclose the information at a certain time. If the information is transmitted to the storage unit 21, and the storage unit 21 assumes that the disclosed information acquired at that one time does not change with time, it is continuously referred to by the drawing unit 22 at subsequent times. good.

Further, when the disclosed information is dynamic information that changes with time, the disclosed information may be transmitted from the disclosure unit 13 to the storage unit 21 at each time. As an example of dynamic information, when a 3D model such as an avatar is used in the disclosed information, the motion of the 3D model may also be included in the disclosed information. In other words, disclosure information may be given at each time, such as when the pause or the like changes with time. The movement of the 3D model may be traced using existing technology (movements of the skeleton and/or facial expressions) of the user, and the traced movement parameters in addition to the information of the 3D model are stored in the storage unit 21. Just send it. Regarding the information of the 3D model drawn by determining the pose etc. according to the motion parameters, it is transmitted as static information to the storage unit 21 only at a certain time, and the motion parameters are used as dynamic information. You may make it transmit to the preservation|save part 21 at each time.

(drawing part 22)
The drawing unit 22 refers to the storage unit 21 that stores measurement information and disclosure information transmitted in real time from each user's information terminal device 10 (in the case of dynamic information), and each user (each own user ), the other user's disclosed information is arranged at a position corresponding to the other user's measurement information (position and orientation information) in the virtual space, and then the other user's disclosed information is arranged. The resulting virtual space is drawn using the user's external parameters (and internal parameters), and the obtained drawing information is transmitted to the corresponding presentation unit 14 of the user's information terminal device 10 .

(presentation part 14)
The presentation unit 14 superimposes the drawing information obtained by the drawing unit 22 on the captured image obtained by the imaging unit 11, thereby obtaining presentation information as a result of adding augmented reality display to the captured image. , presents the presentation information to the user.

Details of the drawing unit 22 and the presentation unit 14 will be further described below.

In one embodiment, the pixel position (u, v) of the drawing information obtained by the drawing unit 22 directly corresponds to each pixel position (u, v) of the captured image captured by the imaging unit 11 of the user. , the area where other user's disclosed information (content) is drawn is composed of the texture of the drawn content, and the area where other user's disclosed information is not drawn is transparent as a transparent area without texture. It can be given in the form of an image. The virtual space used for drawing by the drawing unit 22 uses external parameters of the user and other users and internal parameters of the user to project a three-dimensional virtual space onto a two-dimensional image plane. By drawing the real world imaged by the imaging unit 11, a space is created in which the position and orientation match those of the real world. In this way, in the presentation unit 14, in the same way as in the existing augmented reality display technology that handles a three-dimensional virtual space corresponding to the real world, the image captured by the imaging unit 11 is displayed by the drawing unit 22. By superimposing the drawing information as the obtained transparent image, it is possible to obtain the presentation information in which the augmented reality display is added to the captured image.

The drawing unit 22 refers to the position information in the measurement information among the disclosed information of all the other users stored in the storage unit 21, and this position corresponds to the user's predetermined view volume (visual volume) in the virtual space. Only disclosed information within the range of can be drawn. The user's predetermined view volume may be a cone area (a cone area extending infinitely with the camera position as the apex) within the angle of view range of the camera viewed from the camera position of the user, or the cone Within the area, the distance or depth from the camera position may be within a certain range or less. Here, the angle of view can be obtained from the internal parameters of the camera as described above.

Rendering by the rendering unit 22 is performed on disclosure information of other users positioned within the view volume range. Various embodiments are possible by using the information. Any combination of some or all of these embodiments is also possible.

(1) When arranging other user's disclosure information in the virtual space, after arranging it at the position corresponding to the other user's measurement information (position and orientation information) in the corresponding posture (other user's camera orientation), It may be drawn by the drawing unit 22 . For example, when another user's disclosure information is given as a 3D model and the front orientation is defined, the 3D model is arranged and drawn so that the front orientation matches the other user's camera orientation. can do.

Different from the above, it is also possible to render disclosure information of other users using only position information without using posture information of other users. Information may be used. (This corresponds to the example shown in FIGS. 8 and 9, which will be described later, in which disclosure information BI and CI of other users are drawn facing the user's own camera.)

(2) If the measurement information includes internal parameters acquired by the internal parameter acquisition unit 122, the rendering unit 22 reflects the internal parameters (after setting the viewport range that determines the image range, etc. ,) can be drawn.

(3) When the measurement information includes the depth information acquired by the depth acquisition unit 123 (the depth map that gives the depth at each pixel position (u, v) of the captured image), the drawing unit 22 calculates the After arranging the disclosed information in the virtual space, it is compared with the surface position (spatial position) determined by the depth value in the virtual space defined by the depth map, and the front side of the surface position defined by the depth map (matches on the surface) ) or on the far side (that is, on the shallow side close to the camera or on the deep side far from the camera), and then on the near side (shallow side) The manner in which the disclosure information is drawn may be changed according to whether the disclosure information is on the far side (deep side) or on the deep side.

Here, the disclosed information located on the far side of the surface position defined by the depth map is some kind of obstructing object (for example, a wall, etc.) existing at the surface position defined by the depth map in the real world corresponding to the virtual space. is blocked by Therefore, in one embodiment, disclosure information on the front side of the surface position defined by the depth map is drawn as it is without special processing, while disclosure information on the back side is hidden. Rendering may be performed by performing predetermined processing as a certain object. Rendering that has been processed as being in the occluded state may render the texture in a semi-transparent state, or render the texture so that it is in a completely transparent state (a state that is hidden and cannot be seen). You can choose not to do it. (These schematic examples will be described later with reference to FIG. 9.) Translucent drawing and presentation in which this is superimposed on the presentation unit 14 may be performed using an existing method based on transparency (alpha value). When rendering semi-transparently, the further the disclosure information is located, the more transparent it may be rendered. You can make it

(4) When the measurement information includes the light source information of the user acquired by the light source information acquisition unit 124, the drawing unit 22 reflects the light source information of the user when drawing the disclosed information of other users as a texture. can be drawn using Existing techniques in the 3DCG technical field may be used for rendering reflecting light source information.

8 and 9, schematic examples of the processes described above with respect to the drawing unit 22 and the presentation unit 14 will be shown.

FIG. 8 is a diagram showing a schematic example of a virtual space drawn by the drawing unit 22. As shown in FIG. In FIG. 8, there are information terminal devices 10-A, 10-B, and 10-C as in FIG. 1 described above, and when these exist in the street corner field F1 shown in FIG. A virtual space VS[A] drawn toward the information terminal device 10-A is shown. Therefore, the virtual space VS[A] directly corresponds to the street corner field F1 in the real world.

In FIG _. 8, the information terminal device 10 The camera coordinate system X _A Y _A Z _A of user A's origin P _A is determined from the external parameters of -A. (Here, as is known as the relationship of the viewing pipeline described above, the extrinsic parameter represents the transformation relationship T _A from the world coordinate system XYZ to the camera coordinate system X _A Y _A Z _A , and within the world coordinate system XYZ That is, the position p A expressed by the camera coordinate system X _A Y _A Z _A , which is the position _p in the world coordinate system XYZ, is multiplied by the transformation matrix T _A from the left. Therefore, it can be obtained as "p _A =T _A · p", where the three-dimensional positions p and p _A are given by a four-dimensional vector of homogeneous coordinate representation.) Similarly, for the world coordinate system XYZ, Then, from the external parameters of the information terminal devices 10-B and 10-C, the camera coordinate system X B Y B Z _B of the origin P _B of the user B transformed by the transformation relationship T B and the camera coordinate system X _B Y _B Z _B transformed by the transformation relationship T _C A camera coordinate system X _C Y _C Z _C of the origin P _C of the user C is determined.

Therefore, the transformation relationship from the camera coordinate systems X _B Y _B Z _B and X _C Y _C Z _{C of other users B and C} to the camera coordinate system X _A Y _A Z _A of user A is the product T _A T _B ^-1 and the product T _A T _C ^-1 . (Here, T _B ^-1 and T _C ^-1 are the inverse transforms of T _B and T _C , respectively, from the camera coordinate systems X _B Y _B Z _B and X _C Y _C Z _C of other users B and C to the world coordinates represents the transformation to the system XYZ, for example, the position p in the world coordinate system XYZ to the position p _A expressed in the camera coordinate system X _A Y _A Z _A and the position p _B expressed in the camera coordinate system X _B Y _B Z _B On the other hand, since the relationships ``p <sub>A</sub> = T <sub>A</sub>・p'' and ``p _B =T _B・p'' hold, the relationship ``p _A =T _A T _B ^-1・p _B '' holds, and the above product .) Thus, using the transforming relationship (product T _A T _B ^-1 and product T _A T _C ^-1 ) that can be obtained from the external parameters of each information terminal device 10, As shown in the schematic example, it is possible to arrange and draw disclosure information BI and CI of other users B and C in user A's virtual space VS[A].

In the example of FIG. 8, disclosure information BI and CI are located at positions P _B and P _C of other users B and C, respectively, facing the camera of user A (in the opposite direction of depth direction Z _A which is the front direction of the camera). ) is set as a rectangular signboard with the texts "Disclosure information of B" and "Disclosure information of C" drawn on it, and among the position and orientation information of other users B and C, as the origin of the coordinate system Only the camera positions P _B and P _C are reflected, and the orientation (the orientation of the camera expressed in the orientation of the coordinate axes of the coordinate system) is not reflected. That is, the example of FIG. 8 is an example of the latter among the embodiments of heading (1) of the drawing unit 22 described above. Unlike the example of FIG. 8, the former of the embodiments of heading (1) of the rendering unit 22 described above is applied to render disclosure information BI and CI that also reflects the postures of other users B and C. Also, it is possible by setting the disclosure information to the effect that such drawing should be performed. For example _, the disclosure information BI and CI as text signboards drawn in _FIG . It is also possible to draw in the virtual space VS[A] while facing .

FIG. 9 is a diagram showing schematic examples of drawing by the drawing unit 22 and superimposed display by the presentation unit 14 in the virtual space VS[A] of FIG. Examples EX1 to EX3 each indicate drawing information R1 to R3 and presentation information D1 to D3 obtained by superimposing the drawing information on a captured image (captured image PA in FIG. 5). The information is obtained by drawing and presenting disclosure information BI and CI of other users B and C at the own user's camera position P _A in FIG. 8 in various embodiments.

In the example EX1, the disclosure information BI and CI of the other users B and C are drawn as they are as the drawing information R1, and the presentation information D1 is obtained by superimposing the drawing information R1 on the captured image PA (FIG. 5). ing.

On the other hand, in examples EX2 and EX3, when rendering disclosure information BI and CI of other users B and C as rendering information R2 and R3, depth information ( In this example, the depth information D(PA) in FIG. 7 corresponding to the captured image PA in FIG. 5 is checked to change the rendering mode of the disclosure information positioned deeper than the depth. In both examples EX2 and EX3, the disclosed information BI is determined to be wholly on the nearer side than the depth. It is drawn as BI. On the other hand, in both examples EX2 and EX3, the disclosed information CI is determined to have a part (the central part and the left part of the rectangle) behind the depth. Disclosure information CI-2 and CI-3 drawn in different drawing modes are obtained for a portion on the far side.

Disclosure information CI-2 drawn in example EX2 does not draw any part deeper than the depth (renders completely transparent with 100% transparency) so that drawing information R2 and presentation information D2 are have been obtained. Disclosure information CI-3 rendered in example EX3 is made semi-transparent (rendered in a semi-transparent state with a certain percentage of transparency) behind the depth, and rendering information R3 and presentation information D3 is obtained. In both examples EX2 and EX3, different from example EX1, the disclosed information CI placed at the position of user C is drawn as disclosed information CI-2 and CI-3, thereby creating a wall W1 in the real world (such 9 and 4), the hidden state can be intuitively recognized due to the difference from the disclosed information BI drawn as an unshielded state. As shown in FIG. 4, when user A is a user riding in a vehicle VA, he is hidden behind a wall W1 in front of him at the present time due to the presentation information D2 and D3, but at a time immediately after that, It is possible to effectively draw attention to the presence of user C who is about to appear behind the wall W1.

As described above, according to each embodiment of the present invention, active information sharing among a plurality of information terminal devices 10-A, 10-B, 10-C, etc., that is, the measuring unit 12 in each information terminal device 10 and the information disclosed by the disclosure unit 13 are shared via the server 20, so that the presentation unit 14 of each terminal can display augmented reality as presentation information without the need for image recognition processing of other terminals from the captured image. display can be realized. Therefore, it is necessary to uniquely identify objects that are difficult to image recognize (objects that are invisible due to insufficient light at night, objects that are textureless and are not suitable for detecting local image features, etc.), and objects that are similar to each other and uniquely identified. It is possible to realize augmented reality display even for objects that are difficult to display).

Below, as <1> to <7>, supplementary explanations regarding modified examples and additional examples of the embodiments of the present invention will be given.

<1> Details of Disclosure Information Setting in Disclosure Unit 13 As can be understood from the schematic examples of FIGS. A 3D model that expresses arbitrary content according to the application of augmented reality display in a 3D camera coordinate system with the camera position of the terminal device 10) as the reference (origin) (if it is composed only of planes, lines, and points) including ). For example, _{the disclosure} information CI of the information terminal device 10-C of user _{C in} FIG. Drawing in the camera coordinate system X _A Y _A Z _A (origin P _A ) as the virtual space VS[A] of the device 10-A is also possible. That is, as described above, the 3D model in the camera coordinate system X _C Y _C Z _C (origin P _C ) is transformed by the product T _A T _C ^-1 to the camera coordinate system X _A Y _A Z _A (origin P _A ). can be placed.

Also, in the schematic examples shown in examples EX1 in FIG . 8 and _FIG . , is defined according to the camera position (P _A in FIG. 8) of the own terminal as a reference for drawing, and is defined as accompanying information for drawing accompanying this 3D solid model. Drawing may be performed according to the information. As described above, the drawing unit 22 can perform two types of drawing, the former and the latter described in the embodiment of heading (1).

In one embodiment, the disclosure information as a 3D model may be set so that it is recognized as being present at the camera position of each terminal when it is drawn and displayed in augmented reality. For example, a given point (such as the center of gravity) in the 3D model may be defined to match the camera position of the corresponding terminal. In addition, although the position of the 3D model deviates from the camera position of the corresponding terminal by a certain vector amount, it is recognized as disclosed information linked to the corresponding terminal and its user when displayed in augmented reality. may be defined as For example, on the premise that the terminal is a smartphone and the user stores and holds it in a pocket of clothing or a bag in a predetermined posture, disclosure information as an augmented reality display is drawn near the user (over the head, etc.). As shown, disclosure information can be defined as a 3D solid model in a predetermined vicinity of the camera position. Also, for example, by defining a vector arrow indicating the camera position as its end point in the virtual space and arranging and defining the 3D content at the position of the starting point, the pair of the 3D content and the vector arrow can be used as disclosed information. may be defined. When drawn in the virtual space, the 3D content will be displayed like the annotation indicated by the arrow.

In addition, as shown in the disclosure information CI-2 and CI-3 drawn in the examples EX2 and EX3 of FIG. When drawing disclosure information as, by comparing the depth information with the depth value at each position on the surface of the 3D model (the surface that is visible from the camera that is the reference for drawing the virtual space) , even if it is one 3D model, it is possible to draw distinguishably assuming that part of it is in the occluded state and part of it is not in the occluded state.

<2> Based on a position and orientation different from the position and orientation of the own terminal acquired by the measurement unit 12 (position and orientation acquisition unit 121), the rendering unit 22 renders the virtual space of the own terminal, and the presentation unit 14 renders the virtual space. Presented embodiments are described. This embodiment will be referred to as the second embodiment, and the embodiment described above will be referred to as the first embodiment.

As an example of a suitable application for applying the second embodiment, an avatar that traces the user's motion (skeletal and facial motion) is placed in the virtual space as the user's disclosed information, and through the avatar Applications such as communication in a virtual space can be mentioned. FIG. 10 is a diagram showing a schematic example of arrangement of a virtual space or the like when performing communication or the like using the avatar, as an application example of the second embodiment. Hereinafter, the second embodiment will be described with reference to FIG. can be used arbitrarily.

A field F2 shown in FIG. 10 is assumed to be an entertainment space such as a theater. User B having an information terminal device 10-B is present on the stage ST floor, and information terminal device 10-B is present on the backstage BC floor. There is a user A with A. A wall W2 (a wall W2 that is opaque and blocks light) separates the floors of the stage side ST and the backstage side BC of the information terminal device 10-B and the information terminal device 10-A. The imaging unit 11 is in a state in which it is impossible to capture each other as imaging targets. Therefore, if the first embodiment is applied under the embodiment in which the depth information of heading (3) is used in the drawing unit 22, user A's information terminal device 10-A (camera coordinate system X _A Y _A Z _A , origin P _A ) of user B's information terminal device 10-B (camera coordinate system X _B Y _B Z _B , origin P _B ) is displayed in augmented reality, and vice versa. When the information terminal device 10-A of the user A is displayed in augmented reality in B, and in both cases, the information terminal device 10-A is not drawn or drawn semi-transparently as a state shielded by the wall W2.

In the second embodiment, as shown in FIG. 10, user A's avatar AI is defined as the position and orientation of the information terminal device 10-A, in contrast to the non-rendering or translucent drawing state according to the first embodiment. By being in different positions and orientations (camera coordinate system X _AI Y _{AIZ AI} , origin P _AI ), assuming that user B and his information terminal device 10-B exist on the stage side ST in the same way as user B and his information terminal device 10-B, in the virtual space It is possible to make it appear as disclosed information by user A in . Note that the avatar AI is drawn in the virtual space corresponding to the field F2 in the form of riding on the platform CL as a real object on the stage ST.

Here, if it is assumed that the actual user A and the information terminal device 10-A are present in the position and orientation of the avatar AI of the user A, the first embodiment is applied so that the avatar AI is shielded by the wall W2. Instead, mutual display can be realized between users A and B in the virtual space. (It is assumed that the direction of camera imaging is such that avatar AI is on axis Z _AI , user B is on axis Z _B , and they are facing each other.)

Schematic examples of this display are shown in FIG. 11 as examples EX4 and EX5. (FIG. 11 is a diagram showing a schematic example corresponding to FIG. 10 for explaining the second embodiment, but in the following, first, while explaining the first embodiment based on the above assumption, shall be described with respect to two embodiments.)

In the example EX4, an image P4' (an image P4' that is assumed to have been captured by an actual camera) captured from the avatar AI of the user B side, and user B's disclosure information as drawing information R4 superimposed on this An example of a BI is shown as User B's avatar and the text message "Disclosure information for B" shown above his head. (It should be noted that the information terminal device 10-A and its camera do not actually exist at the position of the avatar AI, and the image P4' is based on the assumption that the camera exists and cannot actually be captured. Therefore, the user B in the image P4' is drawn with a dashed line instead of a solid line.) In the example EX5, the captured image P5 obtained by capturing the side of the avatar AI from the user B is, in the assumed first embodiment, In the case of the second embodiment, the avatar AI exists in the virtual space on the table CL of the stage side ST, but nothing exists in the real world. Then, presentation information D5 obtained by superimposing the avatar AI as drawing information on the captured image P5 is shown. (In the assumed first embodiment, the avatar AI of this presentation information D5 exists as a real object in the real world, and is actually captured in the captured image P5 as well.)

In the second embodiment, the display shown in FIG. 11 (drawing information R4 in example EX4 and presentation information D5 in example EX5) is displayed by the actual user A and the information terminal device 10-A as the position and orientation of the avatar AI (on the stage side). ST), but in the position and orientation shown in FIG. 10 (behind the stage BC). Specifically, the disclosure unit 13, the drawing unit 22, and the presentation unit 14 perform the following processing that is changed from the processing in the first embodiment, thereby realizing the second embodiment. can be done.

Regarding user A who makes an avatar AI with a position and orientation different from the actual position and orientation, the disclosure information setting in the disclosure unit 13 (this setting is transmitted and saved in the storage unit 21 of the server 20) is set as follows. 2 points of information.
(1) The position and _orientation of the _{avatar AI (camera coordinate system X AI Y AI Z AI} , (2) the transformed position and orientation (camera coordinate system X _{AI Y AI} Z _AI , origin P _AI ), _which is schematically indicated by an arrow in FIG. Avatar AI information as a 3D model to be displayed (i.e. disclosure information in the position and orientation transformed by the transformation T _AI )

Here, the above information (2) is similar to setting disclosure information of the 3D model based on the position and orientation of user A (camera coordinate system X _A Y _A Z _A , origin P _A ) in the first embodiment. A similar setting may be performed using the position and orientation (camera coordinate system X _AI Y _AI Z _AI , origin P _AI ) converted by the conversion T _AI as a reference.

It should be noted that, when realizing communication by an avatar, this converted position/orientation (camera coordinate system X _AI Y _AI Z _AI , origin P _AI ) is the viewpoint position/orientation of the avatar AI (simulating a person or the like). You may prepare an avatar AI as a 3D model, which is disclosed information, so as to match the above.

Regarding the other user B who sets the disclosure information BI to a position and orientation matching the actual position and orientation, the setting of the disclosure information in the disclosure unit 13 may be the same as in the first embodiment.

In the drawing unit 22, when drawing the disclosure information (avatar AI) of the user A who has set the avatar AI for the other user B under the conversion T _AI , the information set as described above is stored in the storage unit 21. By referring to the information of (1 ₎ and (2 ₎ , _{the user} A position and orientation (camera coordinate system X _A Y _A Z _A , origin P _A ) converted by transformation T _AI , avatar AI is placed in the position and orientation (camera coordinate system X _AI Y _AI Z _AI , origin P _AI ) and drawn do.

In this way, another user B using the information terminal device 10-B can see his/her own user A as shown in the captured image P5 in the real world, as shown in the example EX5 of FIG. 11 and the arrangement of FIG. Even if it is in an impossible position and posture, it is possible to see the state in which the avatar AI appears in the presentation information D5 as a virtual space.

In the drawing unit 22, when drawing the disclosed information of the other user B for the user A who has set the avatar AI, the position and orientation of the user A (camera coordinate system X _A Y _A Z _A , _{( camera} coordinate _{system X B Y B Z B} , origin P _B ) to place and draw disclosure information.

In this way, the user A using the information terminal device 10-A is in a position and posture in which it is impossible to see the other user B in the real world, as shown in the example EX4 of FIG. 11 and the arrangement of FIG. Also, in the drawing information R5 as a virtual space, it is possible for the user A to see the disclosure information BI of the other user B assuming that the user A is in the position and orientation of the avatar AI.

As described above, in the second embodiment, between users A and B, in a state in which user A has substantially transformed into an avatar AI with a different position and orientation, the motion of user A is reflected in the motion of avatar AI. , communication and the like can be performed in the virtual space.

In the presentation unit 14, the presentation information for the other user B (when the other user B is in the position of the own user) is drawn on the captured image obtained by the imaging unit 11 of the information terminal device 10-B of the other user B. The drawing information drawn by the unit 22 (that is, the avatar AI of the user A who is the other user as seen from the other user B) may be superimposed and presented. At this time, as described above, the disclosure unit 13 allows the user A to include information obtained by tracing the movement of the user A (the movement of the skeleton and/or the facial expression) in the disclosed information, so that the user B can The presented avatar AI can be made to imitate the movement of the user A. On the other hand, in the presentation unit 14, regarding the presentation information for the user A, the captured image obtained by the imaging unit 11 of the information terminal device 10-A of the user A is in the position and orientation of the user A, and the avatar AI is displayed. Since it is not based on the position and orientation, it is sufficient to present only the drawing information drawn by the drawing unit 22 (disclosure information BI of user B) without using this captured image. Alternatively, a 3D model may be defined in advance for the background in the virtual space and stored in the storage unit 21 (for example, the floor of the stage side ST, the wall W2, and the floor behind the stage BC as the background in the field F2 in the real world). etc. is defined as a 3D model and stored in the storage unit 21, and after drawing the modeled background, the drawing information (disclosed by user B) drawn by the drawing unit 22 is drawn on the background. Information BI) may be superimposed and presented.

User A, who uses avatar AI in the second embodiment, obtains depth information of the 3D modeled background from the position and orientation of avatar AI (camera coordinate system X _AI Y _AI Z _AI , origin P _AI ). With this calculation, the drawing unit 22 of the first embodiment may perform drawing similar to the embodiment using the depth information of the headline (3) in the second embodiment.

As described above, in the explanation example of the second embodiment, user A uses an avatar AI that differs from the actual position and orientation, but user B also uses an avatar BI that differs from the actual position and orientation. It is also possible to That is, within the presentation system 100, any part or all of the users may use avatars (disclosure information) that differ from their actual positions and orientations.

Further, in the explanation example of FIG. 10 of the second embodiment, the explanation example assumes that the distance between users A and B is relatively short (for example, about ten meters) only by separating the wall W2. It can be set to any size, and even if users A and B are remote from each other (for example, if they are in different offices several hundred kilometers apart), the second embodiment Applicable. At this time, the conversion T _AI may be set as a predetermined value as information for adjusting the position and orientation of the avatar AI remotely. For example, from the position and orientation of the first marker placed in the first office where user A exists, the position of the second marker (for example, table CL) placed in the remote second office where user B exists If a transformation T _AI is set as a transformation to a posture, when user A is positioned at the first marker, user A's avatar AI will be positioned at the second marker for remote user B. It is possible to present a wide variety of augmented reality displays. Similarly, for user A, it is possible to present user B's disclosure information BI as an augmented reality display near the first marker where the user is located.

As the first marker and the second marker, a square marker or the like in augmented reality display may be used as a marker capable of calculating the position and orientation in the world coordinate system through image recognition or the like. After calculating the position and orientation of the first marker and the position and orientation of the second marker in this way, the transformation T _AI between them may be calculated in advance. (However, when the augmented reality display according to the present invention is performed using the calculated and set conversion T _AI , it is not necessary to image and recognize these markers.)

The second embodiment described above is an example in which the augmented reality display presented to user A is viewed from the viewpoint of avatar AI in a position and orientation different from that of user A (first-person viewpoint of avatar AI). there were. As a modified example of this, the augmented reality display presented to user A is the user A's own viewpoint as in the first embodiment, and the processing that was not realized in the first embodiment is performed by the user's own avatar AI. can be added to the drawing information so that it is viewed from a third-person point of view (a third-person point of view from the point of view of avatar AI, which corresponds to a first-person point of view from the point of view of user A himself). FIG. 12 is a diagram showing a schematic example of a modification of the second embodiment, showing an example EX6 (a modification of example EX4 in FIG. 11) in the same imaging situation between users A and B as shown in FIG. ), a schematic example of a captured image P4'' and drawing information R40 in the information terminal device 10-A of the user A using the avatar AI is shown.

The captured image P4'' is captured with the user A facing the user B, but is blocked by the wall W2 as shown in FIG. CL, user B, etc. are in a state where imaging is not possible. (In FIG. 12, only the positions of the table CL and the user B, which can be imaged when the wall W2 does not exist, are indicated by broken lines on the image P4''.) User A's own avatar AI is drawn at a position on the platform CL in '' and disclosure information BI drawn by another user B. From the third-person viewpoint seen from _PA , the user's own avatar AI (position P _AI ) and other user B's disclosure information BI (position P _B ) can be confirmed as information presented by the presentation unit 14 .

In this modification of the second embodiment, disclosure information of another user B to the user A is drawn in the virtual space of the camera coordinate system X _A Y _A Z _A (origin P _A ) in the same manner as in the first embodiment, and As an additional process in the same virtual space, the user A sets his own avatar AI (which may include information tracing the movement of the user A) and conversion information T _AI as disclosure information in the disclosure unit 13. Then, the avatar AI, which is user A's disclosed information, is placed and drawn at position P _AI (and the posture expressed by the coordinate system X _AI Y _AI Z _AI ) using transformation T _AI , and user A's drawing information This can be achieved by adding the . As in the situation of FIG. 10, when it is known that occlusion will occur, the rendering unit 22 does not use the depth information of the headline (3), and renders the rendering information of the user A and the other user B. may The drawing information of user B (in the case of his own user) may be the same as that of the already described second embodiment, and user B is presented with an augmented reality display as shown in example EX5 in FIG.

<3> The functional block configuration (role sharing) of the information terminal device 10 and the server 20 in the presentation system 100 shown in FIG. 3 is merely an example, and various modifications are possible. For example, only the storage unit 21 may be provided in the server 20 and the drawing unit 22 may be provided in the information terminal device 10. FIG. An embodiment in which both the storage unit 21 and the drawing unit 22 are provided in the information terminal device 10 and the server 20 does not exist is also possible. Information to be saved may be shared. A plurality of information terminal devices 10 may have different configurations. For example, some information terminal devices 10 themselves include the drawing unit 22, while others do not include the drawing unit 22 and entrust the function to the server 20. may

<4> Some of the plurality of information terminal devices 10 in the presentation system 100 may be configured without the disclosure unit 13 . In this case, the information terminal device 10 itself that does not include the disclosure unit 13 can present the disclosed information of the information terminal device 10 as one or more other terminals that include the disclosure unit 13 as an augmented reality display. , when viewed from another terminal, the terminal itself does not disclose the disclosed information, so it is not included in the objects presented as augmented reality display.

<5> As described above, some of the information terminal devices 10 in the presentation system 100 may have the measurement unit 12 and the disclosure unit 13, but may not have the presentation unit 14. . Such an information terminal device 10 plays only the role of making the disclosed information of its own terminal available to other terminals, drawing it, and presenting it as an augmented reality display on the other terminal, and does not perform the augmented reality display on its own terminal.

<6> The presentation unit 14 presents presentation information by superimposing the drawing information obtained by the drawing unit 22 on the captured image obtained by the imaging unit 11. This is based on a video see-through HMD (head-mounted display) as hardware. ), or may be realized using a normal display as a screen of a smartphone or the like. Further, as another embodiment, the presentation unit 14 may present only the drawing information obtained by the drawing unit 22 as the background of the field without using the captured image obtained by the imaging unit 11. In this case, , may be realized using an optical see-through HMD as hardware. As for the optical see-through HMD, calibration is performed in advance, and the same scenery as the scenery of the field in the imaged image obtained by the imaging unit 11 can be seen as the scenery (background) of the real world by the user wearing the HMD. By doing so, the drawing information can be drawn at an appropriate position on the HMD as an augmented reality display.

<7> FIG. 13 is a diagram showing an example of a hardware configuration in a general computer device 70. As shown in FIG. Each of the information terminal device 10 and the server 20 in the presentation system 100 can be implemented as one or more computer devices 70 having such a configuration. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, and one or more dedicated processors 72 (GPU ( graphics processor), deep learning dedicated processor, etc.), RAM 73 as a main storage device that provides a work area to the CPU 71, ROM 74 as an auxiliary storage device, communication interface 75, display 76, mouse, keyboard, touch panel, etc. User input It comprises an input interface 77 for receiving, one or more sensor devices 78, a camera 79 and a bus BS for passing data between them.

Each part of the information terminal device 10 and the server 20 can be implemented by a CPU 71 and/or a dedicated processor 72 that reads and executes a predetermined program corresponding to the function of each part from the ROM 74 . Here, when display-related processing is performed, the display 76 further operates in conjunction, and when communication-related processing relating to data transmission and reception on the network is performed, the communication interface 75 further operates in conjunction. do. A camera 79 and a display 76 constitute the imaging unit 11 and the presentation unit 14 as hardware, respectively. The sensor device 78 is configured as hardware of one or more dedicated sensors that are used when acquiring various measurement information by methods other than analyzing the captured image in the measurement unit 12. The position and orientation acquisition unit 121 GPS, compass, gyro, indoor GPS as a positioning sensor, a LiDAR device as an optical sensor (depth sensor) in the depth acquisition unit 123, an illuminance sensor in the light source information acquisition unit 124, and the like.

In addition, in the information terminal device 10, each hardware as shown in FIG. will match the position and orientation of Therefore, the information about the position and orientation and the surrounding environment obtained based on the output of the camera 79 and the sensor device 78 according to various embodiments represents the information about the position and orientation of the information terminal device 10 and the surrounding environment.

DESCRIPTION OF SYMBOLS 100... Presentation system, 10... Information terminal device, 20... Server, 11... Imaging part, 12... Measurement part, 13... Disclosure part, 14... Presentation part, 21... Storage part, 22... Drawing part

Claims (6)

a measurement unit that measures the position and orientation of itself at each time ;
By referring to position information measured at each time by one or more other terminals and a 3D model configured as an avatar that is disclosure information that is set to be public, the position of itself measured at each time a drawing unit that obtains drawing information by arranging and drawing the one or more 3D models of the other terminal at each time in a three-dimensional virtual space based on the position of the other terminal at the position and predetermined position of the other terminal; ,
a presentation unit that presents the drawing information at each time ,
A reference orientation is defined in the 3D model,
The information terminal, wherein the predetermined posture of the 3D model is a posture in which the 3D model faces the reference direction with respect to the position and posture of the 3D model at each time in the three-dimensional virtual space. Device. In the information terminal device, a predetermined transformation is set for the measured position and orientation of itself,
The rendering unit applies the set predetermined transformation to the measured position and orientation of the user, instead of performing rendering in a three-dimensional virtual space based on the measured position and orientation of the user. 2. The information terminal device according to claim 1 , wherein drawing is performed in a three-dimensional virtual space with reference to the position and orientation of the information terminal device. In the information terminal device, a predetermined transformation of the measured position and orientation of the device itself and disclosure information set to the public regarding the device are set, and
2. The drawing unit further arranges and draws the disclosure information that is set to be open to the public in the position and orientation obtained by applying the predetermined transformation to the measured position and orientation of the user. 2. The information terminal device according to 2 above. At least some of the one or more other terminals are respectively set with a predetermined transformation for the measured position,
In the drawing unit, instead of arranging and drawing the disclosed information of the at least part of the other terminal at the position of the other terminal, a predetermined conversion that is set is applied to the position of the other terminal. 4. The information terminal device according to any one of claims 1 to 3 , wherein drawing is performed by arranging the information terminal device at a position where the image is drawn. 5. The information terminal device according to claim 4 , wherein each of the disclosure information of said at least some of the other terminals is set as an avatar of the user who uses said other terminal. A program that causes a computer to function as the information terminal device according to any one of claims 1 to 5 .

Images