JP7471034B1 - Image processing system and image processing method - Google Patents

Abstract

[Problem] To provide an image processing system and an image processing method that enable the provision of visual data generated using a 3D model without distributing the 3D model. [Solution] The image processing system includes a target terminal that stores a 3D model used to generate visual data related to a target object that can be placed in a virtual space in which unspecified third parties can participate, and an intermediary device that mediates the target object in the virtual space, the intermediary device includes a device-side transmitting unit that transmits to the target terminal address information used in terminal-to-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space, and the target terminal includes a receiving unit that receives viewpoint information including a line-of-sight direction for viewing the target object in the virtual space, a control unit that generates visual data related to the target object in the virtual space based on the viewpoint information, and a transmitting unit that transmits the visual data to the partner terminal by the terminal-to-terminal communication. [Selected Figure] Figure 2

Description

The present invention relates to an image processing system and an image processing method.

In recent years, technology for generating free-viewpoint video on the server side (cloud rendering) has been proposed for generating free-viewpoint video such as 3DoF (Degree of Freedom) and 6DoF (for example, Patent Documents 1 and 2).

Patent Publication No. 2020-515316 Patent Publication No. 2022-532302

As the use of virtual worlds continues to grow, there is also a need among users to be able to show 3D objects to unspecified third parties. Cloud rendering solves this issue, but there is a need to avoid using cloud servers as much as possible or to keep them small-scale due to high costs and other reasons. In addition, due to concerns about the server administrators of cloud servers using 3D objects for unintended purposes or the leakage of 3D objects, it is necessary to consider the needs of users who do not want to use cloud servers in the first place.

The present invention has been made to solve the above-mentioned problems, and aims to provide an image processing system and an image processing method that make it possible to provide visual data generated using a 3D model without distributing the 3D model.

The disclosed embodiment is an image processing system comprising a target terminal that stores a three-dimensional model used to generate visual data related to a target object that may be placed in a virtual space in which unspecified third parties can participate, and an intermediary device that mediates the target object in the virtual space, the intermediary device comprising a device-side transmitting unit that transmits to the target terminal address information used in inter-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space, the target terminal comprising a receiving unit that receives viewpoint information including a line of sight direction in which the target object is viewed in the virtual space, a control unit that generates visual data related to the target object in the virtual space based on the viewpoint information, and a transmitting unit that transmits the visual data to the partner terminal through the inter-terminal communication.

The disclosed aspect is an image processing method used in an image processing system including a target terminal that stores a three-dimensional model used to generate visual data related to a target object that may be placed in a virtual space in which unspecified third parties can participate, and an intermediary device that mediates the target object in the virtual space, the image processing method including the steps of: the intermediary device transmitting to the target terminal address information used for terminal-to-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space; the target terminal receiving viewpoint information including a line-of-sight direction for viewing the target object in the virtual space; the target terminal generating visual data related to the target object in the virtual space based on the viewpoint information; and the target terminal transmitting the visual data to the partner terminal through the terminal-to-terminal communication.

The present invention provides an image processing system and an image processing method that enable the provision of visual data generated using a 3D model without distributing the 3D model.

FIG. 1 is a diagram showing an image processing system 100 according to an embodiment. FIG. 2 is a diagram showing a target terminal 10 according to the embodiment. FIG. 3 is a diagram showing the intermediation server 30 according to the embodiment. FIG. 4 is a diagram for explaining the billboard according to the embodiment. FIG. 5 is a diagram for explaining the billboard according to the embodiment. FIG. 6 is a diagram for explaining the billboard according to the embodiment. FIG. 7 is a diagram for explaining the billboard according to the embodiment. FIG. 8 is a diagram for explaining the billboard according to the embodiment. FIG. 9 is a diagram showing an image processing method according to the embodiment. FIG. 10 is a diagram showing an image processing method according to the second modification.

The following describes the embodiments with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, the drawings are schematic.

[Embodiment]
(Image Processing System)
An image processing system according to an embodiment will be described below. Fig. 1 is a diagram showing an image processing system 100 according to an embodiment.

As shown in FIG. 1, the image processing system 100 has a target terminal 10, a counterpart terminal 20, and an intermediary server 30. The target terminal 10, the counterpart terminal 20, and the intermediary server 30 are connected by a network 110. Although not particularly limited, the network 110 may be configured by the Internet. The network 110 may include a local area network, a mobile communication network, or a VPN (Virtual Private Network).

The target terminal 10 is a terminal used by a user who wants to show a target object (hereinafter, a 3D object). The user may be the owner of a three-dimensional model of the 3D object (hereinafter, a 3D model). The target terminal 10 may be a personal computer, a smartphone, a tablet terminal, or a cloud computer. The target terminal 10 may be any terminal that outputs visual data for displaying a 3D object. The target terminal 10 stores a 3D model of the 3D object. The 3D model may be placed in a virtual space provided by a specific server. The specific server may be the intermediary server 30, or may be a server other than the intermediary server 30 (for example, a server that provides a cloud service that provides a virtual space).

The visual data here is not just data that displays an image, but may also include data such as an image with a depth map, an image with a normal map, an image with bumps and grooves, an image with a height map, a segmented image, or an image that is layered over multiple images. Thus, the visual data does not have to be expressed in an image data format (.png, .jpg, etc.). The visual data may be converted by compression, encryption, etc. The visual data may be used as is, or may be used after additional processing such as lighting, high resolution, or style conversion. The visual data is not the 3D model itself, and the visual data alone cannot restore a 3D model, but it may be data for displaying a rendering result corresponding to a certain viewpoint for a 3D model.

In the embodiment, the target terminal 10 is an example of a target terminal that stores a three-dimensional model used to generate visual data regarding a 3D object that can be placed in a virtual space in which unspecified third parties can participate.

The other terminal 20 is a terminal used by a viewer who requests to view a 3D object. The other terminal 20 may be a personal computer, a smartphone, or a tablet terminal. The other terminal 20 may be a head-mounted display. The other terminal 20 may have a function of displaying a 3D object based on visual data.

In the embodiment, the other terminal 20 is an example of a other terminal that requests viewing of a 3D object in a virtual space.

The intermediary server 30 is a server that mediates between the counterpart terminal 20 and the target terminal 10. The intermediary server 30 may provide a virtual space in which unspecified third parties can participate. The above-mentioned 3D object may be placed in the virtual space. Here, the unspecified third party may mean a third party who has not been authorized by the user of the target terminal 10. The permission may be permission regarding the use of the 3D model of the 3D object. The permission may be indirect permission via the operator of the intermediary server 30 (for example, permission based on a contract between the operator and the user and a contract between the operator and the viewer, etc.).

For example, the intermediary server 30 may solicit viewers (partner terminals 20) who are interested in information about 3D objects by displaying information about the 3D objects in a virtual space. The information about the 3D objects may be a sample of the 3D object. The sample may be a mesh with a reduced resolution of the 3D object, or an image of the 3D object. The information about the 3D object may include a description of the creator of the 3D model of the 3D object, etc.

In the embodiment, the intermediary server 30 is an example of an intermediary device that mediates 3D objects in a virtual space.

(Target devices)
The target terminal 10 according to the embodiment will be described below. Fig. 2 is a diagram showing the target terminal 10 according to the embodiment.

As shown in FIG. 2, the target terminal 10 has a communication unit 11, a storage unit 12, and a control unit 13.

The communication unit 11 may be configured by a communication module. The communication module may be a wireless communication module conforming to a standard such as IEEE802.11a/b/g/n/ac/ax, LTE, 5G, or 6G, or may be a wired communication module conforming to a standard such as IEEE802.3.

The communication unit 11 may communicate with the other terminal 20 and the intermediary server 30. Direct communication between the target terminal 10 and the other terminal 20 may be referred to as inter-terminal communication.

The communication unit 11 may receive address information used in terminal-to-terminal communication from the intermediary server 30. The address information may be the destination of visual data related to a 3D object (e.g., the IP address of the other terminal 20).

The communication unit 11 may receive viewpoint information including a line of sight direction in which a 3D object is viewed in a virtual space. The viewpoint information may include a viewpoint position in which a 3D object is viewed in a virtual space. The communication unit 11 may receive viewpoint information from the other terminal 20.

The communication unit 11 may transmit visual data related to the 3D object to the other terminal 20 through terminal-to-terminal communication. The visual data related to the 3D object may be generated by the control unit 13, which will be described later.

The storage unit 12 is composed of storage media such as a solid state drive (SSD), a hard disk drive (HDD), and a random access memory (RAM), and stores various information.

The storage unit 12 stores 3D models of 3D objects. Although not particularly limited, the type of 3D model may be a wireframe, a surface model, a solid model, etc. The 3D model may be a 3D model that has or can have animation in its appearance or shape. The 3D model may be a 3D model whose appearance or shape can be changed. The 3D model may be a program that can generate an appearance or shape without having the appearance or shape in advance. The 3D model may be a program that can generate visual data corresponding to viewpoint information by a process of interpolating image information from multiple viewpoints, even without having the appearance or shape as explicit data. The 3D model may be a combination of these. Although not particularly limited, the data of the 3D model may be data managed with a specified extension (for example, ".obj", ".ply", ".glb", etc.).

The control unit 13 may include at least one processor. The at least one processor may be configured with a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), one or more Integrated Circuits, one or more Discrete Circuits, or a combination thereof.

The control unit 13 generates visual data regarding the 3D object in the virtual space. Specifically, the control unit 13 generates visual data regarding the 3D object based on viewpoint information received from the other terminal 20.

For example, the control unit 13 may generate visual data for a 3D object as a two-dimensional image in a virtual space based on viewpoint information received from the intermediary server 30. Such a two-dimensional image may be called a billboard. The control unit 13 generates visual data for the billboard using the 3D model stored in the storage unit 12.

Under these conditions, the control unit 13 may perform the following operations:

The control unit 13 may omit sending visual data to the first remote terminal and send visual data to the second remote terminal. The following options are possible for the first remote terminal and the second remote terminal.

In option 1, the first remote terminal may be a remote terminal that does not satisfy a specific condition, and the second remote terminal may be a remote terminal that satisfies a specific condition. The following options are possible specific conditions:

In option 1-1, the specific condition may be winning a lottery. The lottery may be in a form in which the other terminal 20 wins if the random number assigned to the other terminal 20 is a predetermined random number.

In option 1-2, the specific condition may be that the order in which the 3D objects come into view of the viewer of the other terminal 20 in the virtual space is within a predetermined order. The order in which the 3D objects come into view of the viewer of the other terminal 20 may be reset at a predetermined timing (e.g., a certain amount of time has elapsed since the 3D objects were placed in the virtual space, a timing selected by the user of the target terminal 10, etc.).

In options 1-3, the specific condition may be that the billing status of the viewer of the other terminal 20 with respect to the operator of the intermediary server 30 satisfies a condition. For example, the specific condition may be that billing is being performed for viewing a 3D object based on viewpoint information.

In option 2, the first remote terminal may be a remote terminal that transmits a viewpoint position whose distance to a 3D object in virtual space is greater than a threshold, and the second remote terminal may be a remote terminal that transmits a viewpoint position whose distance to a 3D object in virtual space is greater than a threshold.

In other words, for a first partner terminal that transmits a viewpoint position whose distance from the 3D object is greater than a threshold, it is not clear whether the first partner terminal is interested in the 3D object, and it is highly likely that the 3D object does not need to change based on the viewpoint information. Therefore, reducing the load on the inter-terminal communication of the target terminal 10 may be prioritized.

In the embodiment, the communication unit 11 constitutes a receiving unit that receives viewpoint information including the line of sight direction in which a 3D object is viewed in a virtual space. The control unit 13 constitutes a control unit that generates visual data related to the 3D object in the virtual space based on the viewpoint information. The communication unit 11 constitutes a transmitting unit that transmits the visual data to the other terminal 20 through terminal-to-terminal communication.

(Intermediary server)
The intermediation server 30 according to the embodiment will be described below. Fig. 3 is a diagram showing the intermediation server 30 according to the embodiment.

As shown in FIG. 3, the intermediary server 30 has a communication unit 31, a storage unit 32, and a control unit 33.

The communication unit 31 may be configured by a communication module. The communication module may be a wireless communication module conforming to standards such as IEEE802.31a/b/g/n/ac/ax, LTE, 5G, or 6G, or may be a wired communication module conforming to standards such as IEEE802.3.

The communication unit 31 communicates with the target terminal 10 and the other terminal 20. The communication unit 31 may transmit data for displaying the virtual space to the target terminal 10 and the other terminal 20. The communication unit 31 may receive address information of the other terminal 20 from the other terminal 20 requesting viewing of a 3D object. The communication unit 31 may transmit the address information of the other terminal 20 to the target terminal 10 as address information to be used for terminal-to-terminal communication.

The storage unit 32 is composed of storage media such as a solid state drive (SSD) or a hard disk drive (HDD), and stores various information.

The storage unit 32 stores data for displaying the virtual space. The storage unit 32 may manage membership information of the user of the target terminal 10 when membership registration of the user of the target terminal 10 is required. The membership information may include the user's name, the user's handle name, address information of the target terminal 10, etc. Similarly, the storage unit 32 may manage membership information of the viewer of the other terminal 20 when membership registration of the viewer of the other terminal 20 is required. The membership information may include the viewer's name, the viewer's handle name, address information of the other terminal 20, etc.

The control unit 33 may include at least one processor. The at least one processor may be configured with a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), one or more Integrated Circuits, one or more Discrete Circuits, or a combination thereof.

The control unit 33 may instruct the communication unit to transmit address information of the other terminal 20 to the target terminal 10 in response to a request from the other terminal 20.

In the embodiment, the communication unit 31 constitutes a device-side transmission unit that transmits to the target terminal 10 address information used for terminal-to-terminal communication between the other terminal 20 that requests viewing of a 3D object in a virtual space and the target terminal 10.

(Billboard)
The billboard according to the embodiment will be described below. Figures 4 to 8 are diagrams for explaining the billboard according to the embodiment. The billboard may be considered as an example of the above-mentioned visual data, and may be generated based on the above-mentioned visual data.

Here, we will take as an example a case where the 3D object is a human face created using CG (Computer Graphics) or the like, and the human face is superimposed on a human torso in a virtual space. The human torso may be a 3D model that has been generated in advance using CG (Computer Graphics) technology.

For the sake of clarity, the following description will be given taking as an example the viewpoint from which a viewer views a 3D object in a virtual space (hereinafter referred to as the viewer viewpoint). The viewer viewpoint is specified by the viewpoint information described above. The viewer viewpoint may be considered as the viewpoint from which a viewer views a 3D object in a virtual space on the absolute coordinate axes that define the virtual space.

In such a case, at the viewer's viewpoint, the inclination of the billboard may be changed so that the billboard faces directly toward the viewer's viewpoint. That is, the inclination of the billboard is changed based on the viewpoint information. On the other hand, at a viewpoint different from the viewer's viewpoint (hereinafter, the other viewpoint), the inclination of the billboard may be changed based on the relative difference between the viewer's viewpoint and the other viewpoint. The inclination of the billboard may include a tilt about a vertical axis, a tilt about a horizontal axis, or a tilt about a depth axis. The inclination of the billboard may be synchronized with the tilt changed by the viewer's viewpoint. That is, the inclination of the billboard may be changed based on the relative difference between the viewer's viewpoint and the other viewpoint.

Under these assumptions, as shown in Figures 4 to 8, we will illustrate a case in which a billboard 210 including a 3D object 200 is superimposed on a torso 410 in a virtual space 400 in an image from the viewer's viewpoint.

First, as shown in FIG. 4, a case will be described in which the viewer's viewpoint is a viewpoint in which the torso 410 is viewed from the front. The inclination of the billboard 210 is changed so that the billboard 210 faces the viewer's viewpoint directly. On the billboard 210, a display image of the 3D object 200 is generated based on parameters related to the viewer's viewpoint.

Secondly, as shown in FIG. 5, the viewer's viewpoint may be a viewpoint from which the torso 410 is viewed from approximately 30 degrees to the left. The inclination of the billboard 210 is changed so that the billboard 210 faces the viewer's viewpoint directly. On the billboard 210, a display image of the 3D object 200 is generated based on parameters related to the viewer's viewpoint.

Thirdly, as shown in FIG. 6, the viewer's viewpoint may be a viewpoint from which the torso 410 is viewed from approximately 30 degrees to the right. The inclination of the billboard 210 is changed so that the billboard 210 faces the viewer's viewpoint directly. On the billboard 210, a display image of the 3D object 200 is generated based on parameters related to the viewer's viewpoint.

Fourthly, from the viewpoint of explaining the inclination of the billboard 210, an image that can be viewed from a viewpoint different from the viewer's viewpoint will be described. As shown in FIG. 7, when the viewer's viewpoint is a viewpoint from which the torso 410 is viewed from the front, the other viewpoint may be a viewpoint from which the torso 410 is viewed from approximately 30 degrees to the right. The inclination of the billboard 210 is changed so that the billboard 210 faces directly toward the viewer's viewpoint, and therefore, from the other viewpoint, the inclination of the billboard 210 is changed based on the relative difference between the viewer's viewpoint and the other viewpoint.

Fifthly, from the viewpoint of explaining the inclination of the billboard 210, we will explain images that can be viewed from a viewpoint different from the viewer's viewpoint. As shown in FIG. 8, when the viewer's viewpoint is a viewpoint where the torso 410 is viewed from approximately 30 degrees to the left, the virtual space viewpoint may be a viewpoint where the torso 410 is viewed from approximately 30 degrees to the right. The inclination of the billboard 210 is changed so that the billboard 210 faces directly toward the viewer's viewpoint, and therefore, at the other viewpoint, the inclination of the billboard 210 is changed based on the relative difference between the viewer's viewpoint and the other viewpoint.

In Figures 4 to 8, for clarity of explanation, the billboard 210 is shown in the virtual space 400, but the portion of the billboard 210 other than the 3D object 200 may be transparent. In other words, an image of the virtual space 400 may be displayed in the portion other than the 3D object 200.

A billboard may be displayed to one viewer and not to other viewers. There may be one billboard for each viewer.

(Image Processing Method)
An image processing method according to an embodiment will be described below. Fig. 9 is a diagram showing the image processing method according to an embodiment.

As shown in FIG. 9, in step S10, the remote terminal 20 transmits a request to view the 3D object to the intermediary server 30.

In step S11, the intermediary server 30 transmits to the target terminal 10 address information to be used in direct terminal-to-terminal communication between the target terminal 10 and the counterpart terminal 20. The address information may be the IP address of the counterpart terminal 20.

In step S20, the other terminal 20 transmits viewpoint information including the line of sight direction in which the 3D object is viewed in the virtual space to the target terminal 10.

In step S21, the target terminal 10 generates visual data related to a 3D object in a virtual space. The target terminal 10 generates visual data based on viewpoint information using a 3D model stored in the storage unit 12. The visual data related to the 3D object may be visual data that displays the 3D object as a billboard.

In step S22, the target terminal 10 transmits the visual data to the partner terminal 20 via terminal-to-terminal communication.

(Action and Effects)
In the embodiment, the target terminal 10 receives address information of the counterpart terminal 20 from the intermediary server, and transmits visual data to the counterpart terminal 20 through terminal-to-terminal communication. With this configuration, it is possible to provide visual data generated using a 3D model without distributing the 3D model stored in the target terminal 10.

In an embodiment, the target terminal 10 may generate visual data that displays a 3D object as a billboard in a virtual space based on viewpoint information transmitted from the counterpart terminal 20. With this configuration, it is possible to provide visual data generated using a 3D model without passing the 3D model to the counterpart terminal 20.

In an embodiment, the target terminal 10 may omit transmitting visual data to the first partner terminal and transmit visual data to the second partner terminal. With this configuration, there is no need to transmit visual data to all partner terminals 20, and the communication load of terminal-to-terminal communication can be reduced.

[Change Example 1]
Modification 1 of the embodiment will be described below. Differences from the embodiment will be mainly described below.

In modification example 1, in order to reduce the processing load on the target terminal 10, the target terminal 10 may perform the operations shown below.

First, the target terminal 10 may transmit visual data based on representative viewpoint information representative of the two or more viewpoint information to two or more partner terminals transmitting two or more viewpoint information satisfying the first condition. The representative viewpoint information may include a representative gaze direction. The representative gaze direction may be any gaze direction between the two or more gaze directions included in each of the two or more viewpoint information, may be any one of the two or more gaze directions, or may be a midpoint between the two or more gaze directions. The representative viewpoint information may include a representative viewpoint position. The representative viewpoint position may be any gaze position between the two or more viewpoint positions included in each of the two or more viewpoint information, may be any one of the two or more viewpoint positions, or may be a midpoint between the two or more viewpoint positions.

For example, the first condition may be a condition that the angle formed by two or more line of sight directions included in each of the two or more pieces of viewpoint information is within a predetermined angle. The first condition may be a condition that the distance between two or more viewpoint positions included in each of the two or more pieces of viewpoint information is equal to or less than a predetermined distance.

Secondly, the target terminal 10 may transmit visual data based on the specific viewpoint information to one or more specific counterpart terminals that transmit viewpoint information satisfying a second condition for the specific viewpoint information. The specific viewpoint information may include a specific line of sight direction. The specific viewpoint information may include a specific viewpoint position. The specific viewpoint information may be set by the user of the target terminal 10. One or more specific viewpoint information may be set as the specific viewpoint information.

For example, the second condition may be that the angle formed by the line of sight direction included in the viewpoint information and the specific line of sight direction is within a predetermined angle. It may also be that the distance between the viewpoint position included in the viewpoint information and the specific viewpoint position is equal to or less than a predetermined distance.

(Action and Effects)
In a first modification, the target terminal 10 may transmit visual data based on representative viewpoint information that represents the two or more viewpoint information to two or more partner terminals that transmit two or more viewpoint information that satisfy the first condition. With this configuration, a part of the visual data can be made common, thereby reducing the processing load on the target terminal 10.

In the first modification, the target terminal 10 may transmit visual data based on the specific viewpoint information to one or more specific counterpart terminals that transmit viewpoint information that satisfies a second condition for the specific viewpoint information. With this configuration, it is possible to simplify the generation of visual data based on viewpoint information, thereby reducing the processing load on the target terminal 10.

[Change Example 2]
Modification 2 of the embodiment will be described below. Differences from the embodiment will be mainly described below.

In the embodiment, the target terminal 10 generates visual data related to 3D objects for the counterpart terminal 20. In contrast, in Modification Example 2, the intermediary server 30 generates visual data related to 3D objects for some of the counterpart terminals 20 on behalf of some of the target terminals 10. In other words, a case in which some of the target terminals 10 use cloud rendering will be described.

In the second modification, the 3D model stored in the target terminal 10 that uses cloud rendering is also stored in the intermediary server 30. The target terminal 10 that uses cloud rendering may be a target terminal 10 that has been charged for the use of cloud rendering.

The following describes the case where the target terminal 10 is a terminal that is permitted to use cloud rendering.

As shown in FIG. 10, in step S30, the target terminal 10 transmits the 3D model stored in the storage unit 12 to the intermediary server 30.

In step S31, the intermediary server 30 stores the 3D model received from the target terminal 10.

In step S40, the other terminal 20 transmits viewpoint information including the line of sight direction in which the 3D object is viewed in the virtual space to the intermediary server 30.

In step S41, the intermediary server 30 generates visual data for the 3D object based on the viewpoint information using the 3D model stored in step S31. The visual data generated by the intermediary server 30 may be visual data that displays the 3D object as a billboard.

In step S42, the intermediary server 30 transmits visual data regarding the 3D object to the other terminal 20.

(Action and Effects)
In the second modified example, for target terminals 10 using cloud rendering, generation of visual data (cloud rendering) using 3D models stored in the intermediary server 30 is permitted. With such a configuration, although it is necessary to provide 3D models to the intermediary server 30, it is possible to provide visual data generated using 3D models to as many partner terminals 20 as the intermediary server 30's computational resources permit, without the target terminal 10 having to prepare large computational resources, while preventing unlimited outflow of 3D models.

[Change Example 3]
The third modification of the embodiment will be described below. Differences from the embodiment will be mainly described below.

In the third modification, the intermediary server 30 may omit sending address information corresponding to the first partner terminal to the target terminal, and send address information corresponding to the second partner terminal to the target terminal. That is, the intermediary server 30 does not send address information (intermediation process) for the first partner terminal for which the sending of visual data is omitted, and sends address information (intermediation process) for the second partner terminal for which the sending of visual data is executed. As with the embodiment, the following options are possible for the first partner terminal and the second partner terminal.

In option 1, the first counterpart terminal may be a counterpart terminal that does not satisfy the mediation conditions, and the second counterpart terminal may be a counterpart terminal that satisfies the mediation conditions. The mediation conditions may be similar to the specific conditions described above.

In option 2, the first remote terminal may be a remote terminal that transmits a viewpoint position whose distance to a 3D object in virtual space is greater than a threshold, and the second remote terminal may be a remote terminal that transmits a viewpoint position whose distance to a 3D object in virtual space is greater than a threshold.

In other words, for a first partner terminal that transmits a viewpoint position whose distance from the 3D object is greater than a threshold, it is not clear whether the first partner terminal is interested in the 3D object, and it is highly likely that the 3D object does not need to change based on the viewpoint information. Therefore, reducing the load on the inter-terminal communication of the target terminal 10 may be prioritized.

[Change Example 4]
The fourth modification of the embodiment will be described below. The differences from the embodiment will be mainly described below.

In an embodiment, the target terminal 10 may transmit visual data based on representative viewpoint information that represents the two or more viewpoint information to two or more partner terminals that transmit two or more viewpoint information that satisfy the first condition. In Modification Example 4, the operation shown below may be executed as a modification of such an operation.

Specifically, at least one of the two or more counterpart terminals 20 may receive visual data based on the representative viewpoint information from another counterpart terminal 20, instead of receiving visual data based on the representative viewpoint information from the target terminal 10. The other counterpart terminal 20 may be a primary counterpart terminal that receives visual data based on the representative viewpoint information from the target terminal 10, or a secondary counterpart terminal that receives visual data based on the representative viewpoint information from the primary counterpart terminal.

[Change Example 5]
The fifth modification of the embodiment will be described below. The differences from the embodiment will be mainly described below.

In an embodiment, the target terminal 10 may transmit visual data based on the specific viewpoint information to one or more specific counterpart terminals that transmit viewpoint information that satisfies a second condition for the specific viewpoint information. In modification example 5, the operation shown below may be executed as a modification example of such an operation.

Specifically, at least one of the counterpart terminals 20 included in the one or more specific counterpart terminals may receive visual data based on the specific viewpoint information from another counterpart terminal 20, instead of receiving visual data based on the specific viewpoint information from the target terminal 10. The other counterpart terminal 20 may be a primary counterpart terminal that receives visual data based on the specific viewpoint information from the target terminal 10, or a secondary counterpart terminal that receives visual data based on the specific viewpoint information from the primary counterpart terminal.

[Other embodiments]
The present invention has been described by the above disclosure, but the descriptions and drawings forming a part of the above disclosure should not be understood as limiting the present invention. Various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art from this disclosure.

In the above disclosure, the viewpoint information is transmitted from the other terminal 20 to the target terminal 10 by terminal-to-terminal communication. However, the above disclosure is not limited to this. In a case where the address information of the target terminal 10 is not provided to the other terminal 20, the viewpoint information may be transmitted indirectly from the other terminal 20 to the target terminal 10 via the intermediary server 30.

Although not specifically mentioned in the above disclosure, the 3D model stored in the target terminal 10 may be provided to a predetermined counterpart terminal 20. The predetermined counterpart terminal 20 may be a counterpart terminal 20 that is permitted to use the 3D model by the user of the target terminal 10. For example, when the target terminal 10 is requested to view a 3D object by the counterpart terminal 20, the target terminal 10 may transmit data of the 3D model (e.g., data managed by an extension such as ".obj", ".ply", or ".glb") to the counterpart terminal 20. With this configuration, the counterpart terminal 20 can view the 3D model (3D object) in a virtual space without transmitting viewpoint information in the processing after receiving the data of the 3D model. Although the 3D model is handed over to the counterpart terminal 20 that is permitted to use the 3D model, unlimited leakage of the 3D model can be suppressed.

Although not specifically mentioned in the above disclosure, when visual data for displaying a 3D object is not transmitted from the target terminal 10 to the counterpart terminal 20, the counterpart terminal 20 may display a sample of the 3D object. The data for displaying the sample may be transmitted from the target terminal 10 or from the intermediary server 30. The sample may be a reduced-resolution mesh of the 3D object, or an image of the 3D object.

In the above disclosure, an example has been given of a case in which a sample of a 3D object is displayed in a virtual space as information relating to the 3D object when recruiting viewers (partner terminals 20) who are interested in information relating to the 3D object. However, the above disclosure is not limited to this. Information relating to the 3D object does not have to be displayed in a virtual space.

Although not specifically mentioned in the above disclosure, a request to view may include a case where a user actively makes a request, as well as a case where the intermediary server 30 (system) automatically makes a request by monitoring the viewer's viewpoint position, etc. For example, a request to view may be automatically executed when the distance between the display position of the information about the 3D object and the viewer's viewpoint position is equal to or less than a threshold. Alternatively, a request to view may be automatically executed when the viewer's viewpoint position enters an area in the virtual space where a 3D object is placed (e.g., a room, a world, etc.).

Although not specifically mentioned in the above disclosure, omitting the transmission of visual data may mean that visual data displaying a 3D object at a standard frame rate or resolution is not transmitted. For example, even in cases where the transmission of visual data is omitted, visual data displaying a 3D object at a frame rate lower than the standard frame rate or at a resolution lower than the standard resolution may be transmitted.

Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute each process performed by the intermediary server 30. The program may also be recorded on a computer-readable medium. Using a computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or DVD-ROM.

Alternatively, a chip may be provided that is configured with a memory that stores programs for executing each process performed by the intermediary server 30 and a processor that executes the programs stored in the memory.

[Additional Notes]
A first feature is an image processing system comprising: a target terminal that stores a three-dimensional model used to generate visual data related to a target object that may be placed in a virtual space in which unspecified third parties can participate; and an intermediary device that mediates the target object in the virtual space, wherein the intermediary device comprises a device-side transmitting unit that transmits to the target terminal address information used for terminal-to-terminal communication between the target terminal and a partner terminal that requests to view the target object in the virtual space, and the target terminal comprises a receiving unit that receives viewpoint information including a line of sight direction in which the target object is viewed in the virtual space, a control unit that generates visual data related to the target object in the virtual space based on the viewpoint information, and a transmitting unit that transmits the visual data to the partner terminal via the terminal-to-terminal communication.

The second feature is an image processing system according to the first feature, in which the control unit generates, as the visual data, visual data that displays the target object in the virtual space as a two-dimensional image.

The third feature is an image processing system according to the first or second feature, in which the transmission unit omits transmission of the visual data to the first remote terminal and transmits the visual data to the second remote terminal.

The fourth feature is the image processing system according to the third feature, in which the first counterpart terminal is a counterpart terminal that does not satisfy a specific condition, and the second counterpart terminal is a counterpart terminal that satisfies the specific condition.

The fifth feature is an image processing system according to the third or fourth feature, wherein the viewpoint information includes a viewpoint position at which the target object is viewed in the virtual space, the first counterpart terminal is the counterpart terminal that transmits the viewpoint position at a distance from the target object in the virtual space that is greater than a threshold value, and the second counterpart terminal is the counterpart terminal that transmits the viewpoint position at a distance from the target object in the virtual space that is greater than the threshold value.

The sixth feature is an image processing system in which, in at least one of the first to fifth features, the transmission unit transmits the visual data based on representative viewpoint information that represents the two or more viewpoint information to two or more partner terminals that transmit two or more viewpoint information that satisfy a first condition.

The seventh feature is an image processing system in which, in at least one of the first to sixth features, the transmission unit transmits the visual data based on the specific viewpoint information to one or more specific partner terminals that transmit viewpoint information that satisfies a second condition for the specific viewpoint information.

The eighth feature is an image processing system in which, in at least one of the first to seventh features, the device-side transmitting unit omits transmitting the address information corresponding to the first remote terminal to the target terminal, and transmits the address information corresponding to the second remote terminal to the target terminal.

The ninth feature is the image processing system according to the eighth feature, in which the first counterpart terminal is a counterpart terminal that does not satisfy the mediation conditions, and the second counterpart terminal is a counterpart terminal that satisfies the mediation conditions.

The tenth feature is an image processing system according to the eighth or ninth feature, wherein the viewpoint information includes a viewpoint position at which the target object is viewed in the virtual space, the first counterpart terminal is the counterpart terminal that transmits the viewpoint position at a distance from the target object in the virtual space that is greater than a threshold value, and the second counterpart terminal is the counterpart terminal that transmits the viewpoint position at a distance from the target object in the virtual space that is greater than the threshold value.

The eleventh feature is an image processing method used in an image processing system including a target terminal that stores a three-dimensional model used to generate visual data related to a target object that may be placed in a virtual space in which an unspecified third party can participate, and an intermediary device that mediates the target object in the virtual space, the image processing method including the steps of: the intermediary device transmitting to the target terminal address information used for terminal-to-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space; the target terminal receiving viewpoint information including a line-of-sight direction for viewing the target object in the virtual space; the target terminal generating visual data related to the target object in the virtual space based on the viewpoint information; and the target terminal transmitting the visual data to the partner terminal through the terminal-to-terminal communication.

10: Target terminal, 11: Communication unit, 12: Storage unit, 13: Control unit, 20: Partner terminal, 30: Intermediary server, 31: Communication unit, 32: Storage unit, 33: Control unit, 100: Image processing system, 110: Network, 200: 3D object, 210: Billboard, 400: Virtual space, 410: Torso

Claims (12)

A target terminal that stores a three-dimensional model used to generate visual data related to a target object that can be placed in a virtual space in which an unspecified third party can participate;
an intermediary device that mediates the target object in the virtual space,
The intermediary device provides the virtual space viewable by the other terminal,
The target terminal is
a receiving unit that receives viewpoint information including a line of sight direction in which the target object is viewed in the virtual space;
a control unit that generates visual data regarding the target object in the virtual space based on the viewpoint information;
a transmission unit that transmits the visual data to the other terminal by terminal-to-terminal communication between the other terminal and the target terminal . 2. The image processing system according to claim 1, wherein the intermediate device includes a device-side transmitting unit that transmits address information corresponding to the other terminal to the target terminal as address information used for the terminal-to-terminal communication. The image processing system according to claim 1, wherein the control unit generates, as the visual data, visual data that displays the target object in the virtual space as a two-dimensional image. The image processing system according to claim 1, wherein the transmission unit omits transmission of the visual data to the first remote terminal and executes transmission of the visual data to the second remote terminal. The first counterpart terminal is the counterpart terminal that does not satisfy a specific condition,
The image processing system according to claim 4 , wherein the second counterpart terminal is the counterpart terminal that satisfies the specific condition. the viewpoint information includes a viewpoint position from which the target object is viewed in the virtual space,
the first counterpart terminal is a counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is greater than a threshold value;
The image processing system according to claim 4 , wherein the second counterpart terminal is the counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is closer than the threshold value. The image processing system according to claim 1, wherein the transmission unit transmits the visual data based on representative viewpoint information that represents two or more pieces of viewpoint information to two or more other terminals that transmit the two or more pieces of viewpoint information that satisfy a first condition. The image processing system according to claim 1, wherein the transmission unit transmits the visual data based on the specific viewpoint information to one or more specific partner terminals that transmit viewpoint information that satisfies a second condition for the specific viewpoint information. 3. The image processing system according to claim 2 , wherein the device-side transmitting unit omits transmitting the address information corresponding to a first remote terminal to the target terminal, and transmits the address information corresponding to a second remote terminal to the target terminal. The first opposite terminal is the opposite terminal that does not satisfy the mediation condition,
The image processing system according to claim 9 , wherein the second counterpart terminal is the counterpart terminal that satisfies the mediation condition. the viewpoint information includes a viewpoint position from which the target object is viewed in the virtual space,
the first counterpart terminal is a counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is greater than a threshold value;
The image processing system according to claim 9 , wherein the second counterpart terminal is the counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is closer than the threshold value. An image processing method for use in an image processing system including a target terminal that stores a three-dimensional model used to generate visual data relating to a target object that may be placed in a virtual space in which an unspecified third party can participate, and an intermediary device that mediates the target object in the virtual space, comprising:
A step in which the intermediary device provides the virtual space viewable by a counterpart terminal ;
receiving, by the target terminal, viewpoint information including a line of sight direction in which the target object is viewed in the virtual space;
generating visual data for the target object in the virtual space based on the viewpoint information by the target terminal;
The image processing method includes a step of the target terminal transmitting the visual data to the other terminal by terminal-to-terminal communication between the other terminal and the target terminal .