JP7280223B2 - Information terminal device, remote communication support method and program - Google Patents

Description

The present invention relates to an information terminal device, a remote communication support method, and a program for supporting remote communication.

There is a teleconferencing system technology that enhances the sense of presence in remote communication. For example, Patent Literature 1 discloses a system in which each user stands with their backs to the display, and images of the user together with the display captured by a camera with a polarizing film installed are displayed on the display of the other party. The surroundings you see and the surroundings you see are the same, the sense of direction and distance are symmetrical, and the situation can be observed from a third party. claims to be able to produce Patent Literature 2 discloses a system that superimposes an avatar to which recognized gestures and facial expressions are applied to a self-propelled robot to which markers are attached. It claims that by reflecting the user's facial expressions and gestures on the avatar, it is possible to communicate almost face-to-face.

JP-A-2000-122767 Patent No. 6564484

However, the conventional techniques as described above have the following problems.

Patent Document 1 has a problem that the movable range of the user is limited. For example, the user cannot leave the angle of view of the camera. In addition, there is a problem that changes in the positional relationship with the other party due to movement are not reflected. For example, since the other party is only displayed on the display, the side of the other party cannot be seen even if the user moves left or right. Furthermore, there is a problem that it is impossible to make eye contact with the other party. For example, since the display and the camera need to face each other, if the user tries to see the other party displayed on the display, the back of the user is displayed on the other party's display.

Patent Literature 2 solves part of the above problem. For example, the viewpoint position can be freely changed by moving the robot. However, there is a problem that the self-propelled robot needs to be prepared in a remote location in advance, and the device becomes large-scale. In addition, it is premised that there is a clear distinction between the viewer and the viewer, and if we were to eliminate this distinction and achieve two-way communication, it would be necessary to prepare robots for both sides. Such a case is not assumed. If two-way communication is to be realized in this way, the image from the robot prepared for the other party and the image superimposed on the avatar of the other robot in front of the other party will be displayed on both sides. , and a sense of realism cannot be obtained.

As described above, in the prior art, remote communication with presence could not be realized efficiently. As an existing method that can be used to realize remote communication with a sense of presence, there is an augmented reality display that is also used in Patent Document 2, but a simple augmented reality display is placed in the environment as in Patent Document 2. If the avatar of the other party is not used, for example, the avatar of the other party is displayed in an immovable area in the environment of one's own side (for example, the avatar of the other party is displayed in a state where the avatar of the other party is buried inside a wall. avatar is displayed), and such a phenomenon can impair the sense of realism in remote communication (for example, even if you try to move yourself close to the other party's avatar, Since the avatar exists inside the wall (on the back side), it is blocked by the wall and cannot move), but the conventional technology could not deal with this. Furthermore, even if one recognizes an immovable area in one's own environment, it is not considered in the prior art to efficiently display the avatar of the other party in order to realize remote communication with a sense of reality under the restrictions. was not

SUMMARY OF THE INVENTION In view of the above problems of the prior art, it is an object of the present invention to provide an information terminal device, a remote communication support method, and a program capable of supporting efficient realization of realistic remote communication.

In order to achieve the above object, the present invention provides an information terminal device that is used by the own user and supports remote communication between the own user and other users, wherein the position and orientation of the own user are used as first positioning information. a measuring unit that acquires first region information as an estimate of the user's movable range; and second state information of the other user from the other user, and the position of the other user and second positioning information as an orientation, and second area information as an estimate of the movable range of the other user, and in the virtual camera viewpoint set to the position and orientation of the first positioning information, 2. The other user's avatar or the other user reflecting the position and orientation of the positioning information and the second state information is drawn, and the coordinates of the second area information are transformed so as to approximate the first area information. a drawing unit that draws at least part of the converted second area information as representing a recommended range of movement of the user; and a presentation unit that presents the result drawn by the drawing unit to the user. It is characterized by having

Also, the present invention is a remote communication support method between a first user and a second user, which is executed by at least one device, comprising: obtaining the position and orientation of the first user and the second user as first positioning information and second positioning information, respectively; and movement of the first user and the second user. Obtaining first area information and second area information as an estimated possible range, and obtaining the position and orientation of the second positioning information and the position and orientation of the second positioning information at the virtual camera viewpoint set to the position and orientation of the first positioning information. Transformed second area information obtained by drawing the second user's avatar or the second user reflecting the second state information and coordinate-transforming the second area information so as to approximate the first area information Rendering at least a part of the image as representing a recommended range of movement of the first user, presenting the rendering result to the first user; Rendering the avatar of the first user or the first user reflecting the position and orientation of the first positioning information and the first appearance information from a camera viewpoint, and approximating the second area information. drawing at least part of the transformed first region information obtained by coordinate-transforming the first region information as representing a recommended range of movement of the second user, and presenting the drawing result to the second user; , is provided. Further, the present invention is characterized by a program that causes a computer to function as the information terminal device.

According to the present invention, it is possible to support remote communication with a sense of presence by drawing the avatar of the other user or the other user in the virtual space based on the virtual camera arrangement set as that of the own user. By drawing a recommended range of movement shared between the user and other users by coordinate conversion in the virtual space, it is possible to support remote communication with a sense of realism and efficiency.

1 is a configuration diagram of an information system according to one embodiment; FIG. 1 is a functional block diagram of an information terminal device according to an embodiment; FIG. FIG. 4 is a functional block diagram showing the flow of processing between two information terminal devices. FIG. 5 is a diagram showing a schematic example of drawing processing by a drawing unit; FIG. 10 is a diagram showing schematic examples in which a movement non-recommended area, which is not desirable to be set wide, can be set as narrow as possible by coordinate conversion. 4 is a flow chart of the operation of the information terminal device in real time according to one embodiment. FIG. 10 is a diagram showing a schematic example of drawing only a portion or a boundary portion thereof in a drawing unit; It is a figure which shows the example of the hardware constitutions in a common computer.

FIG. 1 is a configuration diagram of an information system according to one embodiment. The information system 100 includes at least information terminal devices 10A and 10B used by users A and B who remotely communicate with each other. The information terminal devices 10A and 10B are configured to be able to communicate with each other via the network NW as indicated by the bidirectional arrow L0. Users A and B can perform remote communication using the devices 10A and 10B.

The information system 100 further includes a server 20, so that various information can be transmitted and received between the information terminal devices 10A and 10B via the server 20, as indicated by the two-way arrows L1 and L2. remote communication can also be realized. When the information system 100 is configured only by the information terminal devices 10A and 10B (referred to as the first embodiment), part of the processing that needs to be performed in the information terminal devices 10A and 10B is performed when the server 20 is further provided (the second embodiment). Second embodiment), the server 20 may be responsible.

In the following, after the first embodiment in which the server 20 does not exist and only the information terminal devices 10A and 10B exist, the processing that the server 20 can perform in the second embodiment in which the server 20 also exists will also be described. explain.

The information terminal devices 10A and 10B may be configured as mobile terminals such as smartphones or head-mounted displays (HMD) carried or worn by the users A and B, or may be configured in an environment where the users A and B exist (for example, a conference room). It may be configured as a device that is fixedly installed in the device, or it may be configured as a combination of these. For example, the information terminal devices 10A and 10B may be configured as a combination of a camera installed in a conference room and a smartphone capable of local communication with the camera. may differ between the information terminal devices 10A and 10B.

FIG. 2 is a functional block diagram of the information terminal device 10 according to one embodiment. In FIG. 2, the information terminal device 10 includes a recognition unit 1, a positioning unit 2, a measurement unit 3, a drawing unit 4 and a presentation unit 5. Here, the configuration common to the information terminal devices 10A and 10B shown in FIG. 1 is the configuration of the information terminal device 10 shown in FIG. (As described above, the information terminal devices 10A and 10B may have different hardware configurations, but may have the same functional configuration as shown in FIG. 2.)

FIG. 3 shows the flow of processing between two information terminal devices 10 as a functional block diagram. Assuming the common configuration shown in FIG. 2, information transfer between the information terminal devices 10A and 10B will be described as appropriate with reference to FIG. At this time, mutual processing is common between the user A and B information terminal devices 10A and 10B, and the information to be handled is symmetrical. , and user B is another user and uses the information terminal device 10B as another terminal. (That is, the same explanation holds even if users A and B are interchanged.)

3, and as described above, the information terminal devices 10A and 10B used by users A and B have the same configuration as the information terminal device 10 in FIG. In order to distinguish them, the reference numerals with A and B at the end of each of the functional units 1 to 5 in FIG. 2 are used. For example, the recognition unit 1 of the information terminal device 10 is referred to as a recognition unit 1A in the user A's information terminal device 10A.

The processing of each part of the information terminal device 10 of FIG. 2 is as follows.

By recognizing a user who uses the information terminal device 10, the recognition unit 1 acquires state information representing the state of the user, outputs this state information to the drawing unit 4 of the own terminal, and This state information is transmitted to the drawing unit 4 of the other terminal. (In FIG. 3, the recognizing unit 1A of user A outputs the obtained state information of user A to its own drawing unit 4A and also transmits it to the drawing unit 4B of other user B. Note that FIG. 2 omits the flow of sending status information to other terminals.)

The state information about the own user recognized and acquired by the recognition unit 1 can be acquired as visual information such as facial expression, posture (pose), and information related to all or part of gestures, and can be acquired using any existing method. It is possible to recognize such state information. As an existing method, for example, the method of Non-Patent Document 1 below may be used for posture recognition. Any existing method may be used for facial expression recognition and gesture recognition as well.
Non-Patent Document 1: Adrian Bulat et al., ``Toward fast and accurate human pose estimation via soft-gated skip connections,'' ArXiv, 2002.11098v1, 2020.

Recognition unit 1 can obtain state information by processing an image of the user using an existing method. Alternatively, a camera or the like placed in the environment where the user is present (for example, a conference room) may be used. At this time, a passive or active marker attached to the user's body or terminal may be used together to improve the recognition accuracy of the state information.

The positioning unit 2 outputs the measured coordinates and orientation of the own user as positioning information to the drawing unit 4 of the own user, and also transmits this positioning information to the drawing unit 4 of the other terminal. (In FIG. 3, the positioning unit 2A of own user A outputs the acquired positioning information of own user A to its own drawing unit 4A and also transmits it to the drawing unit 4B of other user B. Note that FIG. 2 omits the flow of transmitting positioning information to other terminals.)

The positioning unit 2 can acquire positioning information using any existing method. For example, a positioning sensor such as a GPS (Global Positioning System), a compass, or a gyro that is standard equipment in a mobile terminal such as a portable terminal may be used as hardware for realizing the positioning unit 2 . As a method for realizing the positioning unit 2, the surrounding environment of the user may be captured by a camera, and positioning information may be obtained as the position and orientation of the camera by image matching of known coordinates for this image. The positioning information may be obtained using triangulation or the like by the station. These methods are based on the premise that the mobile terminal is carried or worn by the user as a part or all of the information terminal device 10, and the mobile terminal (or the positioning sensor mounted in the mobile terminal) is used. etc.) is obtained as the positioning information of the own user, but the positioning information may be obtained by directly recognizing the own user. For example, one or more cameras placed in the environment may image the user and obtain positioning information. Existing methods such as SLAM (Self-Position Estimation and Environment Map Creation) based on motion parallax and stereo vision can be used for camera positioning. Also, any combination of the existing methods as described above can be used.

The measurement unit 3 measures the surrounding environment of the user, estimates an area where the user can move, which is an area other than the area where the user cannot move due to obstacles such as walls and fixtures, and automatically estimates the area as area information. It outputs to the drawing unit 4 of the user, and also transmits this area information to the drawing unit 4 of the other terminal. (In FIG. 3, the user A's measuring unit 3A outputs the acquired region information of the user's A surrounding environment to its own drawing unit 4A and also transmits it to the other user B's drawing unit 4B. Note that the flow of transmitting area information to other terminals is omitted in FIG.

A LiDAR (lidar, light detection and ranging) sensor, depth camera, etc. may be used for the hardware that realizes the measurement unit 3, and in the form of 3D point cloud data, depth map, etc., the position of the sensor or camera can be used. The position where the surface of some object appears for the first time when looking in various directions (the back side of the position (far side from the camera) is an area where the user cannot move, and the front side of the position is an area where the user can move. area information can be obtained as data recording the As a method of obtaining area information as point cloud data or a depth map from a LiDAR sensor or a depth camera, any existing method based on stereo vision or the like may be used.

In addition, when the positioning unit 2 is realized by the SLAM method, the positioning information by the positioning unit 2 is acquired as a result of self-position estimation by SLAM, and the environment map is also obtained in the form of 3D point cloud data by SLAM at the same time. Since the environmental map is created, the environmental map may be used as the area information measured by the measuring unit 3. FIG. That is, the positioning unit 2 and the measurement unit 3 are realized by SLAM, the self-position estimation result is adopted as the positioning information acquired by the positioning unit 2, and the created environment map is adopted as the area information acquired by the measurement unit 3. can be

The drawing unit 4 uses the state information, the positioning information, and the area information obtained by the recognition unit 1, the positioning unit 2, and the measurement unit 3 of both the own terminal and the other terminal as described above, respectively, to determine the environment of the own user. By arranging and drawing the other user's avatar (alternate character) in the virtual space superimposed on the , drawing information is obtained and output to the presentation unit 5 of the own terminal. The presentation unit 5 displays and presents the drawing information to the user, so that the user can visually recognize the state of the other user in the form of an avatar placed in the virtual space, and remotely communicate with the other user. It becomes possible to communicate.

Specifically, the drawing unit 4 (1) defines a virtual space and arranges the other user's avatar, and (2) draws the other user's avatar in the virtual space by reflecting the appearance of the other user. (3) Rendering information is generated by rendering a movement non-recommended area in the virtual space as a representation of an area that should not be moved in the virtual space. (For the sake of explanation, the codes are explicitly distinguished such as own user A, other user B, etc.)

(1) The drawing unit 4A of user A uses the position and direction in the positioning information acquired by the positioning unit 2A of user A as the position and direction of the virtual camera for defining the coordinates of the virtual space for drawing. In this virtual space, the position and orientation in the positioning information acquired and transmitted by the positioning unit 2B of other user B are the viewpoint of the avatar of other user B (that is, the position of the face in the avatar and the front of the face). The avatar of the other user B is drawn assuming that the direction of the line of sight as the direction) is arranged.

Note that the positioning unit 2A of user A acquires positioning information defined in the three-dimensional world coordinates of the environment in which user A exists, thereby creating a virtual space for drawing another user B's avatar. The coordinates can be set as those that match the three-dimensional world coordinates of the environment in which user A exists (that is, the real world). As a result, the drawing unit 4A of the user A can draw the avatar of the other user B as an object to be displayed in augmented reality in the environment in which the user A exists.

In addition, in order to arrange the position and orientation in the positioning information of the avatar of other user B in the virtual space of own user A, the coordinate transformation T (from the coordinates of other user B to the Coordinate transformation T) to coordinates can be used. That is, by applying the coordinate transformation T to the positioning information (assumed to be L(B)) defined in the three-dimensional world coordinates of the environment in which the other user B exists in the positioning unit 2B of the other user B, Transformed positioning information T(L(B)) arranged in user A's virtual space may be calculated.

(2) After the virtual space is defined by the processing of (1) above and the viewpoint of the avatar of the other user B is arranged in this virtual space, the drawing unit 4A of the user A recognizes the other user B. The other user B's avatar is drawn in this virtual space by reflecting the state information of the other user B recognized and transmitted by the unit 1B.

That is, if the state information includes a facial expression (for example, a facial expression defined by the open/closed state of the eyes, the position of the iris, the orientation of the eyebrows, the open/closed state of the mouth, etc.), the drawing unit 4A assumes that the facial expression is included. , to draw another user B's avatar. Also, if the state information includes a posture, the drawing unit 4A draws the avatar of the other user B assuming that the posture is taken. For example, if the posture is given by skeletal joint position information (skeleton information), the avatar of the other user B may be drawn with the skeletal joint positions given by the skeleton information. Also, if the state information includes a gesture, the drawing unit 4A may draw the avatar of the other user B assuming that the gesture is performed.

Regarding the avatar of other user B, model data of the avatar is set in advance by other user B in the form of a three-dimensional CG (computer graphics) model of a human being, etc., and various types of data are stored in this model data. By reflecting the situation information, the drawing unit 4A can draw the avatar of the other user B. FIG. Any existing method may be used for drawing the avatar reflecting the state information.

(3) In the drawing unit 4A of user A, in the virtual space set in the processing of (1) above, a region where user A should not move (even if it is possible to move in the environment of the real world) as a moving non-recommended region. Specifically, in the drawing unit 4A, the area information (assumed to be R(B)) acquired and transmitted by the measurement unit 3B of the other user B is coordinate-transformed (area information R(B) is The coordinate-transformed area information R _A (B) is placed in the virtual space, and the external area of the coordinate-transformed area information R _A (B) is drawn as a movement-non-recommended area. be able to.

Note that the user A is located inside the coordinate-transformed area information R _A (B) in the virtual space in which the augmented reality display _is performed. The movement non-recommended area may be represented by drawing only the position of the surface to be represented. User A sees area information R _A (B) drawn as a surface in the virtual space, and thus an area that is farther from the surface as viewed from user A is displayed using area information R _A (B). can be recognized as an external area, that is, a movement non-recommended area. It should be noted that any specific texture or the like that is prepared in advance may be used for drawing as the surface.

Note that the non-recommended movement area has the effect of making remote communication more natural by acting so that the user does not move into the area, and enhancing the sense of presence. Examples are described below.

Coordinate transformation (referred to as coordinate transformation T) for obtaining area information R A (B) for arranging in the virtual space from area information R( _B ) of other user B in the drawing unit 4A is the measurement of user A itself. It is possible to calculate a coordinate transformation T such that "T(R(B))≈R(A)", where R(A) is the area information obtained by the unit 3A. That is, the area information T(R(B)) obtained by subjecting the area information R(B) to the coordinate transformation T matches the area information R(A) as much as possible so that the area information T(R(B) ) and the area information R(A) (that is, so as to approximate the area information R(A) by minimizing the transformation error). As a result of applying , area information R _A (B)=T(R(B)) for placement in the virtual space can be obtained. (Note that even if R(A) is given by user A's own three-dimensional world coordinates WLA and R(B) is given by user B's own three-dimensional world coordinates WLB, WLA and WLB are the same. Given that WLA and WLB are given in a common world coordinate system, it is not necessary to consider them to be the same (without having to find the transformation between WLA and WLB). In the same way, if the

Note that the non-recommended movement area drawn and represented by the area information R _A (B) obtained by the coordinate transformation T indicates that the user A should not move into the area. will limit the In other words, although the non-recommended movement area has the effect of enhancing the presence of remote communication (described later in FIG. 4), the wider the non-recommended movement area, the narrower the range in which user A can move freely. It may become a situation that impairs the convenience of However, by calculating the area information R _A (B) using the coordinate transformation T, it is possible to minimize the range of such movement-unrecommended areas, and to ensure the convenience of remote communication. Become. This will be explained later with reference to the schematic example of FIG.

Note that the coordinate transformation T may be calculated by any existing method such as numerical calculation that minimizes transformation errors as a combination of all or part of translation transformation, rotation transformation, and enlargement/reduction transformation. For example, when the region information is given as 3D point cloud data, the coordinate transformation T is calculated as including translational transformation and rotational transformation by applying point cloud registration by any existing method. Alternatively, for example, the method of Non-Patent Document 2 below may be used as a point group registration method that also allows scaling conversion.
Non-Patent Document 2: O. Hirose, ``A Bayesian Formulation of Coherent Point Drift,'' IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.

Note that when the coordinate transformation T accompanies enlargement/reduction transformation, the amount of movement in the virtual space of the avatar drawn in the virtual space is also automatically scaled. For example, if another user B moves 1m in the environment (in the real world), the avatar of the other user B drawn and displayed in the virtual space of user A moves twice as much as 2m in the virtual space. It is possible to do so.

FIG. 4 is a diagram showing a schematic example of drawing processing by the drawing unit 4 described above. In FIG. 4, as an example EX11, an environment SA in which the information terminal device 10A of the user A is arranged and an environment SB in which the information terminal device 10B of the user B is arranged. It is shown as a model example assuming the case of being indoors. In user A's environment SA, area information R(A) is acquired as a model of indoor walls WA and placed furniture EA. In user B's environment SB, area information R(B) is acquired as a model of indoor walls WB and placed furniture EB.

In the example EX12 of FIG. 4, under the environments SA and SB of the above example EX11, the renderings by the rendering units 4A and 4B are displayed on the presentation units 5A and 5B, respectively, to provide the users A and B with virtual images. Schematic examples of spaces VA and VB are shown.

In the virtual space VA, the avatar BR of the user B on the other side is drawn by augmented reality display for the user A, the wall WA, and the furniture EA existing in the environment SA as the real world, and the area information of the user B is displayed. A surface EBR corresponding to the surface area of the fixture EB in the environment SB is drawn as an expression of the movement non-recommended area determined by the area information R _A (B) obtained by coordinate transformation of R(B). Similarly, in the virtual space VB, the avatar AR of the user A on the other side is drawn by augmented reality display for the user B, the wall WB, and the furniture EB existing in the environment SB as the real world. A surface EAR corresponding to the surface area of the fixture EA in the environment SA is drawn as an expression of the movement non-recommended area determined by the area information R _B (A) obtained by coordinate-transforming the area information R(A) of the environment SA.

In the virtual space VA, the user A can move in the indoor area behind the surface EBR within the environment SA as the real world, but movement is not recommended for remote communication with the user B via the avatar BR. Recognize it as being an area. Assuming that user A moves into this non-recommended movement area, in user B's virtual space VB on the other side, user A's avatar AR is assumed to be buried inside fixtures EB. , a situation occurs in which the presence of the remote communication is lost due to the unnatural display. In the present embodiment, the user A is made to recognize the movement non-recommended area as the surface EBR, and is urged not to move beyond it, thereby making it possible to avoid such a situation that impairs the presence.

A similar effect can be obtained in the virtual space VB by interchanging "A, B" in the above description. In other words, if user B moves to the back of the surface EAR, from the standpoint of user A, the avatar BR of the other user B will be displayed buried in the furniture EA, thereby impairing the sense of presence. However, from the user B's point of view, the environment SB It is possible to encourage limiting the range of movement within.

As described above, as shown in the schematic example of FIG. 4, the drawing unit 4 draws the non-recommended movement area to enhance the sense of realism in remote communication. Therefore, from the viewpoint of convenience of remote communication, etc., it is not desirable to set a wide movement non-recommended area. That is, in the example of FIG. 4, when the movement range is two-dimensionally expressed on the floor (ground), the user A has a real movement range MA within the environment SA, but the non-recommended movement area is excluded from this. User B is prompted to move only within the recommended movement range LMA limited by . You are prompted to move only within the recommended movement range LMB.

FIG. 5 shows that a coordinate transformation T calculated by an appropriate method is used to set a non-recommended movement area that is not recommended to be widened as narrow as possible (in other words, it is a complementary set of the non-recommended movement area. FIG. 10 is a diagram showing a model example in which it is possible to set the recommended movement range as wide as possible, divided into examples EX1 to EX4.

In the example EX1 of FIG. 5, it is assumed that the area information is given in two dimensions as a schematic example, and the two-dimensional area information in a checkerboard pattern that provides a bird's-eye view of the situation in which users A and B are in different rooms is used as data DA and DB. The white areas indicate movable areas, and the black areas indicate areas that cannot be moved due to walls, fixtures, etc. The same applies to the example in FIG. In the following examples EX2 to EX4, a wide recommended movement range is set by various calculated conversion processing T (conversion processing T equivalent to planar alignment between data DA and DB) based on the situation of this example EX1. It shows that it is possible.

In the virtual space of example EX2, user A and user B are adjacent to each other as shown in data DAB2 as a result of setting the transformation processing T to be limited to translation in the horizontal direction only in the figure. It shows a case where the conversion processing T is calculated as a translational movement. A gray area indicates an area that cannot be moved in the environment of either user A or B, and the same applies to the example in FIG. As shown in the data DA2, in user A's virtual space, movement restrictions are urged in gray areas so that user A does not enter user B's immovable area with respect to user A's movable area. ing. As shown in the data DB2, in the user B's virtual space, a part of the movable area is also an obstacle area of the user A's environment, and the movement is restricted in a gray area.

The virtual space of data DAB3, DA3, and DB3 of example EX3 expands the calculation range of conversion processing T than example EX2, and horizontal and vertical translations in the figure (horizontal and vertical combinations combine diagonal translation ), and as a result of setting to calculate the conversion processing T, the conversion processing T is calculated as a translation such that the users A and B are obliquely arranged. Compared to example EX2, it can be confirmed that movement restrictions are relaxed and the movable area is widened. Furthermore, the virtual space of data DAB4, DA4, and DB4 of example EX4 expands the calculation range of conversion processing T than example EX3, and horizontal and vertical translation and rotation (90°, 180°, 270° As a result of setting to calculate the conversion process T by limiting it to a rotation of °), the environment of user A is rotated 90 degrees clockwise, and the translation is such that user A and user B are adjacent to each other. A case where the conversion processing T is calculated is shown. Comparing with Examples EX2 and EX3, it can be confirmed that the movement restrictions are further relaxed and the movable area is further widened.

That is, in the order of data DA2, DA3, and DA4, the number of white square areas corresponding to the size of the area in which user A is allowed to move within his own virtual space is 5, 7, and 8. increasing with each. Similarly, in the order of the data DB2, DB3, and DB4, the number of white square areas corresponding to the size of the area in which user B is allowed to move within his own virtual space is 5, 7, increased to 8. As shown in these examples, in the present embodiment, users A and B have the same shape of regions in which users A and B are allowed to move within their respective virtual spaces. In the example of FIG. 5, the sizes of the data DA and DB (horizontal x vertical) are 3×4 and 4×4, respectively. Although it is not, if it is set so that scaling conversion is also allowed from example EX4 and the conversion processing T is calculated, the movement allowable range in the virtual space is further expanded.

As shown in the example of FIG. 5 above, by applying the coordinate transformation T that is set as a combination of all or part of translation, rotation, and scale transformation, each user A and B reflects the real environment. The degree of freedom of communicating with the avatar of the other user while moving around in the virtual space can be maximized within the range set by the coordinate transformation T, and the effect of minimizing movement restrictions is achieved. can get.

As described above, according to the information terminal devices 10A and 10B of the present embodiment, support for remote communication with a sense of presence between users A and B can be provided in a manner that is comfortable for both users A and B with few restrictions on movement and a high degree of freedom. can be realized in the form

Various supplementary matters and additional matters will be described below.

<1> In the above explanation, it is assumed that the time is fixed to one time. 10B can support remote communication between users A and B in real time.

FIG. 6 is a flowchart of real-time operation of the information terminal device 10 according to one embodiment. In step S1, it is determined whether or not the current time t has reached a predetermined processing timing. If the answer is affirmative (if the processing timing has been reached), the process proceeds to step S2, and if the answer is negative (if the processing timing has not reached). returns to step S1 and waits in step S1 until an affirmative determination is obtained. The determination of the processing timing in step S1 may be performed at a predetermined processing rate, for example, at a timing that matches a predetermined frame rate for displaying the drawing information in the presentation unit 5. FIG.

In step S2, the recognition unit 1, the positioning unit 2, and the measurement unit 3 perform recognition, positioning, and measurement processing, respectively, and acquire state information, positioning information, and area information about the own terminal/user at current time t. Then proceed to step S3. In step S3, state information, positioning information, and area information about the own terminal and own user at the current time t acquired in step S2 are transmitted to another terminal (as the other party of remote communication) in real time in the same manner as the own terminal. ), and state information, positioning information, and area information about the other terminal/other user at the current time t acquired in step S2 and transmitted in step S3 in the other terminal After receiving it in its own terminal, it proceeds to step S4.

In step S4, state information, positioning information, and area information about the own terminal/user at current time t acquired in step S2, and state information and positioning information about other terminals/users received from other terminals in step S3. And the drawing unit 4 draws using the area information, and the drawing result is presented to the user by the presentation unit 5. Then, the process returns to step S1, and at the next current time t+1, which is the next processing timing, Similar processing is continued.

<2> In the embodiment of FIG. 6, the measurement unit 3 measures area information in real time, and the drawing unit 4 uses this to draw. On the premise that the area information as the estimated area does not change with time, the measurement unit 3 measures the area information in advance in each user's environment in advance, and the drawing unit 4 draws fixed area information that does not change with time. It may be used in That is, only the recognizing unit 1, the positioning unit 2, the drawing unit 4, and the presentation unit 5, excluding the measuring unit 3, may perform real-time processing in the information terminal device 10. FIG.

When the area information of both the own user and other users is measured in advance, the above-mentioned coordinate transformation T can also be calculated in advance as a fixed value that does not depend on the time t, and the drawing unit 4 can use it for drawing. can.

<3> In the above description, the measurement unit 3 measures the area in which the user can move as area information. Since the movable area and the immovable area have the same information, the area information may be measured as the immovable area. Exactly similarly, information representing the boundary between an immovable area and a movable area corresponding to the same contents may be measured as area information.

<4> In exactly the same way as the definition of the area information measured by the measurement unit 3 above, the rendering unit 4 renders the movement-non-recommended area in the virtual space in the above description, but it is synonymous with the complementary set. The recommended movement area may be drawn, or the boundary surface between the non-recommended movement area and the recommended movement area may be drawn. In addition, the drawing may be performed in any drawing mode that allows the user to understand that the drawing result is the non-recommended movement area/recommended movement area/these boundary surfaces. For example, the rendering unit 4 may preliminarily set an agreement that the boundary surface should be a semi-transparent red surface and render it in the three-dimensional virtual space, and notify the user of the agreement through a document such as a user manual. You can leave it. Alternatively, a text description may be rendered in the virtual space to inform the user of the agreement. In this way, the user perceives the movement-non-recommended area/movement-recommended area by confirming in the presentation unit 5 that a translucent red surface that does not exist in the real environment is drawn in the virtual space as the drawing information. It becomes possible to

In other words, for the purpose of realizing remote communication with a sense of reality as described with reference to FIGS. 4 and 5, it is against the purpose of the user to move into the non-recommended movement area, and should be avoided. The user is informed in advance that it is recommended to move only within the recommended movement area, and any of the non-recommended movement area/recommended movement area/these boundary planes is drawn in an arbitrary drawing mode. You can draw either

<5> When the drawing unit 4A of user A draws area information R _A (B) that corresponds to the movable range of other user B and expresses that user A should not move beyond that range. may draw only a portion of the area information R _A (B) instead of the entire area (or only the boundary represented by the portion). Specifically, of the entire area information R _A (B), a product set "RA ( _B )∩R(A)" that overlaps with the area information R(A) as the movable range of the user A ”, or only the difference set “R _A (B)\R(A)” representing the same content as a complementary set, or the intersection or difference set of these It is also possible to draw only the boundary part of

FIG. 7 is a diagram showing a schematic example of drawing only a portion or a boundary portion thereof as described above. In FIG. 7, when area information is given as three-dimensional information, examples of this area information and corresponding drawing are shown in a two-dimensional cross section as a schematic example. Area information R(A) of own user A and transformed area information R _A (B) of other user B are shown as a result of calculation of coordinate transformation T (not shown). (Regional information R(A) and R _A (B) are two-dimensional cross-sections of three-dimensional information, both of which have a square shape, and users A and B can move within this square in each environment (room, etc.). ) Examples EX21 to EX24 are examples of drawing shown in a two-dimensional cross section as schematic examples.

As the drawing by the drawing unit 4A for this example EX20, the intersection "R _A (B)∩R(A)" or the complement "R _A (B) \ R(A)” may be drawn as the movable area and non-movable area of user A, respectively. (In addition, the complement "R _A (B) \ R (A)" is composed of four triangular regions in example EX22.) Also, as corresponding to the same content, it is shown in thick lines in example EX23. A boundary portion BD1 such as this may be drawn as the boundary between the movable area and the immovable area, or a boundary portion BD2 (a part of the boundary portion BD1 and four A boundary portion BD2 (=“BD1∩R(A)”) inside the area information R(A) of user A may be drawn as a line segment.

From the standpoint of user A, the area information R(A) is the range in which user A can actually move in the environment in which he or she is placed (the real world such as a room). Since the area information R _A (B) obtained by converting the range of movement permitted from the standpoint displayed to the other user B, the product set "R _A (B)∩R (A)” is the allowable movement range, but as a mode of expressing this in virtual space (virtual space superimposed on the real world), use any of examples EX21, E22, EX23, EX24, etc. good too. (In the schematic example of FIG. 4, the complementary set "R _A (B)\R(A)" of example EX22 in FIG. 7 and the boundary part BD2 of example EX24 are drawn in the virtual space VA of own user A. It corresponds to the surface EBR (equivalent to the surface area of the fixture EB of the environment SB) as an unmovable location for the other user B.)

Note that FIG. 7 shows an example from the standpoint of user A, but the same explanation holds for other user B by replacing the signs of A and B. FIG. In particular, the shape of the range in which users A and B are allowed to move in the virtual space is the intersection "R _A (B)∩R(A)" (or the intersection "R _B (A)∩R(B) ”), users A and B can share a common virtual space while existing in different environments (different places in the real world), enabling remote communication with a sense of reality. (As described above, the scale may change due to scaling conversion.) Note that the virtual spaces shared by users A and B are five, seven, and eight in the example of FIG. are shown as individual white grid areas.

<6> Using the state information recognized by the recognition unit 1, the drawing unit 4 of the remote communication partner draws an avatar (a user's alter ego character) that reflects the state information. Alternatively, or in addition, a photographed image of the user may be drawn in the virtual space as a billboard (a two-dimensional signboard on which the photographed image is arranged) as a so-called "real avatar". When billboard drawing is performed, the state information recognized by the recognition unit 1 may be information obtained by cutting out only the user's area from a real image captured by a camera placed in the environment. The billboard should be drawn at the position and orientation given by the user's positioning information. Any existing method such as object recognition using deep learning or the like may be used to cut out the user area from the photographed image. The billboard may consist of the user's whole body, or may consist of only the face area. A billboard only for the face area and an avatar for the body part below the face may be used in combination.

<7> As hardware for realizing the presentation unit 5, a video see-through HMD or an optical see-through HMD may be used. When using a video see-through HMD, the drawing information drawn by the drawing unit 4 (other user avatars and recommended movement regions) are superimposed on each other and presented by the presentation unit 5. FIG. When using an optical see-through HMD, as described above, by presenting only the drawing information drawn by the drawing unit 4 in the presentation unit 5, the augmented reality display described in FIG. 4 etc. can be presented to the user.

<8> In addition to recognizing state information of the own user, the recognition unit 1 performs recording in the environment of the own user. It is also possible to make it possible for both parties to have a conversation in remote communication by playing back the voice in real time.

<9> When the measurement unit 3 measures the area information as the area where the user can move/impossible, for example, after calculating the area information as a point cloud, image recognition is applied to the image corresponding to the point cloud. Then, by deleting a part of the point cloud that corresponds to a predetermined type of object that can be moved automatically or manually (for example, a chair, etc.) from the entire point cloud, the final area information may be defined. A predetermined type of movable object may be drawn using a predetermined three-dimensional model in the partner's virtual space together with the avatar.

<10> In the above description, the case of remote communication between two users A and B was taken as an example, but it is also possible to perform remote communication with three or more users. For example, when a third user C further exists and remote communication is performed between users A, B, and C using the information terminal device 10C, self-user A coordinates region information alignment with other user B. Transformation T _A is performed, and furthermore, region information is aligned with another user C by coordinate transformation T _C , and the avatars and the like of other users B and C are drawn in the virtual space. Also, two or more users may exist in one remote environment. For example, if three users A, B1, and B2 exist, and user A exists in a first environment and users B1 and B2 exist in a second environment, users B1 and B2 have common area information. It should be treated as a thing.

<11> After obtaining area information as three-dimensional coordinate information such as a point group or depth map, the measurement unit 3 may quantize the area information as voxels to speed up the calculation of the coordinate transformation T and the like. When the coordinate transformation T is calculated by the rendering unit 4, the rotation transformation may be limited to the rotation transformation around the vertical direction in the three-dimensional world coordinates. As shown in the schematic example of FIG. 5, the measuring unit 3 may obtain area information as two-dimensional coordinate information corresponding to the horizontal plane of the three-dimensional world coordinates. When the area information is obtained as two-dimensional coordinate information, and the area information is quantized and given in a grid as shown in the schematic example of FIG. It may be set in advance, and the coordinate transformation T may be calculated by an exhaustive search in all candidates, like image template matching.

<12> FIG. 8 is a diagram showing an example of a hardware configuration in a general computer device 70. As shown in FIG. Each of the information terminal devices 10A, 10B and the server 20 in the information system 100 can be implemented as one or more computer devices 70 having such a configuration. When each of the information terminal devices 10A and 10B and the server 20 is implemented by two or more computer devices 70, information necessary for processing may be transmitted and received via a network. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, and a GPU (graphics processing unit) as a dedicated processor that executes part or all of the execution instructions of the CPU 71 instead of the CPU 71 or in cooperation with the CPU 71. ) 72, RAM 73 as a main storage device that provides a work area to CPU 71 (and GPU 72), ROM 74 as an auxiliary storage device, communication interface 75, display 76, input interface 77 that accepts user input by mouse, keyboard, touch panel, etc. It includes a camera 78 that captures an image of the environment and a user, one or more sensors 79 that perform sensing and measurement using a method other than image capturing, such as a LiDAR sensor, and a bus BS for exchanging data between them.

Each functional unit of the information terminal devices 10A, 10B and the server 20 can be realized by the CPU 71 and/or the GPU 72 that reads and executes a predetermined program corresponding to the function of each unit from the ROM 74 . Both the CPU 71 and the GPU 72 are a kind of arithmetic unit (processor). Here, when display-related processing is performed, the display 76 further operates in conjunction, and when communication-related processing relating to data transmission/reception is performed, the communication interface 75 further operates in conjunction. The presentation unit 5 may be implemented as the display 76 to output an augmented reality display.

When the server 20 is used as the second embodiment described above, the server 20 may have the function of the drawing unit 4, for example, but acquires information that can only be acquired in the environment of the information terminal devices 10A and 10B. The server 20 may be responsible for all or part of arbitrary processing other than the processing.

10, 10A, 10B... information terminal device, 1, 1A, 1B... recognition section, 2, 2A, 2B... positioning section, 3, 3A, 3B... measurement section, 4, 4A, 4B... drawing section, 5, 5A, 5B…Present part

Claims (9)

An information terminal device that is used by the user and supports remote communication between the user and other users,
a positioning unit that acquires the position and orientation of the own user as first positioning information;
a measurement unit that acquires first region information as an estimate of the user's movable range;
receiving from the other user second state information of the other user, second positioning information as the position and orientation of the other user, and second region information as an estimated movable range of the other user; ,
Rendering the other user's avatar or the other user reflecting the position and orientation of the second positioning information and the second state information at a virtual camera viewpoint set to the position and orientation of the first positioning information; and a drawing unit that draws at least part of the transformed second area information obtained by coordinate-transforming the second area information so as to approximate the first area information as representing a recommended range of movement of the user;
an information terminal device, comprising: a presentation unit that presents a result drawn by the drawing unit to the user. The first positioning information, the second positioning information, and the second state information are obtained in real time, and the first area information and the second area information are obtained as predetermined information measured in advance. The information terminal device according to claim 1. 3. The information terminal device according to claim 1, wherein said coordinate transformation is configured as a combination of all or part of translation transformation, rotation transformation and scaling transformation. 4. The information terminal device according to claim 1, wherein said drawing unit performs said coordinate transformation so as to maximize overlap between said transformed second area information and said first area information. . The state information of the other user includes an image of the other user,
5. The information terminal device according to any one of claims 1 to 4, wherein an image of the other user is used when the drawing unit draws the other user. 6. The information terminal device according to any one of claims 1 to 5, wherein the second state information is acquired as information relating to all or part of facial expressions, postures, and gestures of the other user. 7. The rendering unit according to any one of claims 1 to 6, wherein the rendering unit renders an overlapping portion of the converted second area information and the first area information as a portion corresponding to a recommended range in which the user can move. The information terminal device according to 1. A method of supporting remote communication between a first user and a second user, performed by at least one device, comprising:
Acquiring the first state information of the first user and the second state information of the second user;
obtaining the positions and orientations of the first user and the second user as first positioning information and second positioning information, respectively;
Acquiring first area information and second area information as estimated movable ranges of the first user and the second user, respectively;
drawing the second user's avatar or the second user reflecting the position and orientation of the second positioning information and the second state information at a virtual camera viewpoint set to the position and orientation of the first positioning information; and drawing at least a part of the transformed second area information obtained by coordinate-transforming the second area information so as to approximate the first area information as representing a recommended range of movement of the first user, presenting the drawing result to the first user;
drawing the first user's avatar or the first user reflecting the position and orientation of the first positioning information and the first state information at a virtual camera viewpoint set to the position and orientation of the second positioning information; and drawing at least part of the transformed first area information obtained by coordinate-transforming the first area information so as to approximate the second area information as representing a recommended range of movement of the second user, and presenting the drawing result to the second user. A program that causes a computer to function as the information terminal device according to any one of claims 1 to 7.

Images