WO2024029275A1

WO2024029275A1 - Display control system

Info

Publication number: WO2024029275A1
Application number: PCT/JP2023/025317
Authority: WO
Inventors: 志織安江
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2022-08-02
Filing date: 2023-07-07
Publication date: 2024-02-08

Abstract

A server 10 according to one embodiment of the present invention is provided with: a file storage unit 13 that stores, for a first virtual object, a first file having a first file size and a second file having a second file size larger than the first file size; and a display control unit 14 that controls display of the first virtual object. The display control unit 14 executes: a first process for displaying the first file on a display unit 24 by preferentially transmitting the first file to a user terminal 20; a second process for transmitting the second file to the user terminal 20 after the transmission of the first file or in parallel with the transmission of the first file; and a third process for, when the first file is being displayed on the display unit 24 and the transmission of the second file is completed, switching the first file being displayed on the display unit 24 to the second file.

Description

display control system

One aspect of the present invention relates to a display control system that controls the display of virtual objects.

Technologies for presenting virtual objects (virtual objects) to users, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) (hereinafter referred to as "XR technology"), are known. For example, Patent Document 1 discloses a mechanism for providing augmented reality or mixed reality to a user by displaying a virtual object superimposed on a real space image displayed on a display device.

Japanese Patent Application Publication No. 2014-203382

In the above-mentioned XR technology, from the viewpoint of increasing the comfort of the user's virtual experience, it is required to speed up the display of virtual objects as much as possible. For example, as a system applying the above-mentioned XR technology, there is a system in which a virtual object, which is a 3D object generated by scanning an object existing in real space, is downloaded and displayed on a user terminal. However, such virtual objects based on real objects generally have large file sizes, so it takes time to download them, and it may take a long time for the virtual objects to be displayed on the display of the user terminal. be. On the other hand, if the file size of a virtual object is reduced by reduction etc. in order to shorten the download time, the quality of the user's virtual experience will deteriorate because the virtual object cannot be delivered to the user at its original quality (resolution). Another problem may arise:

Therefore, one aspect of the present invention is to provide a display control system that can maintain the quality of the user's virtual experience while shortening the time it takes for a virtual object to be displayed on a user terminal. do.

A display control system according to one aspect of the present invention is a display control system that performs display control of a virtual object displayed on a display unit of a user terminal, and the display control system controls the display of a first virtual object, which is a first virtual object, in a first file. a file storage unit that stores a first file having a file size and a second file having a second file size larger than the first file size; and a display control unit that controls display of the first virtual object on a user terminal. , is provided. The display control unit performs a first process of displaying a first virtual object based on the first file on the display unit by transmitting the first file to the user terminal with priority over the second file, and transmitting the first file. a second process of transmitting a second file to the user terminal after or in parallel with transmitting the first file; and a first virtual object based on the first file is displayed on the display unit; performing a third process of switching the first virtual object based on the first file displayed on the display unit to the first virtual object based on the second file when transmission of the second file to has been completed; do.

In the display control system according to one aspect of the present disclosure, first, a first file having a small file size (first file size) is preferentially transmitted to the user terminal. With the above configuration, it is possible to shorten the time until the first virtual object is displayed on the display unit of the user terminal. Further, after the first file is transmitted, or in parallel with the first file transmission, the second file is transmitted to the user terminal. After the user terminal receives the second file, the first virtual object based on the first file displayed on the display unit is switched to the first virtual object based on the second file. With the above configuration, the first virtual object based on the second file, which has a larger file size and higher quality than the first file, can be presented to the user later, so that the quality of the user's virtual experience can be maintained.

According to one aspect of the present invention, it is possible to provide a display control system that can maintain the quality of a user's virtual experience while reducing the time it takes for a virtual object to be displayed on a user terminal.

FIG. 1 is a diagram illustrating an example of a functional configuration of a system including a server and a user terminal, which is a display control system according to an embodiment. FIG. 3 is a diagram showing an example of file generation processing. FIG. 3 is a diagram schematically showing an example of display control of a virtual object. FIG. 2 is a sequence diagram illustrating an example of operations of a server and a user terminal. It is a diagram showing an example of the hardware configuration of a server and a user terminal.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the accompanying drawings. In addition, in the description of the drawings, the same or equivalent elements are given the same reference numerals, and redundant description will be omitted.

FIG. 1 is a diagram showing an example of a system including a server 10, which is a display control system according to an embodiment, and a user terminal 20. The server 10 is a device that controls the display of virtual objects displayed on the display unit 24 of the user terminal 20. For example, the server 10 provides the user with an AR experience or an MR experience by displaying a virtual object together with a real space image on the display unit 24 of the user terminal 20. Alternatively, the server 10 may provide the user with a VR experience by displaying a virtual object together with a virtual background (virtual space image) on the display unit 24 of the user terminal 20. The user terminal 20 is a computer terminal used by a user. The type of user terminal 20 is not limited. For example, the user terminal 20 may be a personal computer such as a desktop PC or a laptop PC, or may be a terminal device such as a smartphone or tablet held in the user's hand, or may be a terminal device such as a user's head or the like. It may also be a wearable device that is worn on a part of the body. Examples of wearable devices include goggle-type, glass-type (glasses-type), hat-type, and other head-mounted displays.

The server 10 and the user terminal 20 are connected to each other so as to be able to communicate data via a wired or wireless communication network. Note that the server 10 and the user terminal 20 may be connected via one or more other devices. For example, when the user terminal 20 is a head-mounted display, each of the server 10 and the user terminal 20 may be configured to be able to communicate with other devices such as the user's smartphone. In this case, the server 10 is configured to be able to communicate with the user terminal 20 via the other device.

As shown in FIG. 1, the server 10 (display control system) includes a reception section 11 (terminal information acquisition section), a file generation section 12, a file storage section 13, and a display control section 14. Further, the user terminal 20 includes an imaging section 21, a transmitting section 22, a receiving section 23, and a display section 24.

First, each functional configuration of the user terminal 20 will be explained. The imaging unit 21 has a function of capturing an image of the real space around (for example, in front of) the user terminal 20 to obtain a real space image (imaging data). The imaging unit 21 is configured by, for example, a camera. The real space image may be a still image or a moving image (video). When an AR experience or an MR experience is provided to a user, the real space image may be displayed on the display unit 24. In addition, in order to generate a virtual object that is a 3D object by the file generation unit 12, which will be described later, the imaging data is a plurality of images obtained by capturing images of an object (real object) existing in real space from a plurality of different directions. It may also be composed of image data.

The transmitting unit 22 transmits the imaging data acquired by the imaging unit 21 to the server 10. Further, in this embodiment, the transmitter 22 transmits terminal information regarding the attributes of the user terminal 20 to the server 10. Examples of attributes of the user terminal 20 include application type, communication speed, processing performance, and the like.

"Application type" is information indicating the type of application used to display the virtual object sent from the server 10. The application type can be specified, for example, by a combination of the type of user terminal 20 and the type of application running on the user terminal 20. Examples of application types include a dedicated application that runs on a PC, a web browser that runs on a PC, a web browser that runs on a smartphone, and an application built into a head-mounted display such as smart glasses. The transmitter 22 transmits terminal information indicating the application type to the server 10, for example, when starting execution of a service that displays a virtual object transmitted from the server 10 (for example, during a login operation).

"Communication speed" is the download speed (download speed) when downloading data from the server 10 to the user terminal 20. For example, when the user terminal 20 is a terminal that moves with the user (e.g., a head-mounted display, a smartphone, etc.) and the server 10 and the user terminal 20 communicate wirelessly, the strength of the radio waves changes depending on the location of the user terminal 20. Therefore, the communication speed may change. For example, the transmitter 22 periodically transmits terminal information indicating the communication speed to the server 10.

"Processing performance" is information indicating the specifications of the processor, memory, etc. that the user terminal 20 has, the processor usage rate, the memory usage rate, etc. For example, terminal information indicating processing performance at a certain point in time is information indicating the amount of computational resources (processor, memory, etc.) available on the user terminal 20 at that point in time. For example, the transmitter 22 periodically transmits terminal information indicating processing performance to the server 10.

The receiving unit 23 receives information for displaying a virtual object on the display unit 24 (a first file, a second file, etc. to be described later) from the server 10.

The display unit 24 displays a virtual object in a display area 24a provided in the display unit 24 based on instructions from the display control unit 14 of the server 10. Specifically, the display unit 24 displays the virtual object in the display area 24a based on information for displaying the virtual object (first file, second file, etc.) received from the server 10. When the user terminal 20 is a PC, the display unit 24 may be configured by a display monitor. When the user terminal 20 is a mobile terminal such as a smartphone, the display section 24 may be configured with a touch panel display. When the user terminal 20 is a head-mounted display, the display unit 24 may be configured by a display placed in front of both eyes of the user.

Next, each functional configuration of the server 10 will be explained. The receiving unit 11 receives various data transmitted by the user terminal 20 (transmitting unit 22). In this embodiment, the receiving unit 11 receives imaging data acquired by the imaging unit 21. The receiving unit 11 also functions as a terminal information acquisition unit that acquires the above-mentioned terminal information (application type, communication speed, processing performance, etc.).

The file generation unit 12 acquires the imaging data received by the reception unit 11. When the imaging data includes a real object that is an object existing in real space, the file generation unit 12 generates a first file and a second file of a virtual object corresponding to the real object based on the imaging data. do. The file generation unit 12 corresponds to a real object included in the imaging data as a subject, for example, by performing at least one of image recognition (2D image recognition), photogrammetry, and 3D scanning on the imaging data. Generate a file representing a virtual object. Note that the method of generating a virtual object by the file generation unit 12 is not limited to the generation method based on real imaging data as described above. For example, the file generation unit 12 may generate a file representing a virtual object by modeling or the like. That is, the virtual objects handled in this embodiment are not limited to objects based on real objects, but may be objects created by creators such as CG designers.

An example of file generation processing by the file generation unit 12 will be described with reference to FIG. 2. FIG. 2 shows an example in which a rabbit kept as a pet by a user is imaged as a real object by the user terminal 20 (imaging unit 21). In this example, the file generation unit 12 generates three types of files 50A, 50B, and 50C as data representing a virtual object (first virtual object) corresponding to this real object. The files 50A, 50B, and 50C are all data used to display a virtual object on the display unit 24 of the user terminal 20, and may include drawing information for drawing the virtual object in the display area 24a. In this embodiment, the virtual object is expressed as a 3D object.

The file 50A is data that has not been compressed (reduced) and retains its original quality (resolution). That is, the file 50A is an original file generated based on imaging data of a real object. The file 50B is data obtained by compressing the file 50A by 30%. That is, the file size (file capacity) of the file 50B is 70% of the file size of the file 50A. The file 50C is data obtained by compressing the file 50A by 60%. That is, the file size of the file 50C is 40% of the file size of the file 50A.

Files 50B and 50C have lower quality than file 50A due to the data compression described above. As an example, as shown in FIG. 2, the surface shape of the virtual object (rabbit) represented by the file 50B is rougher than the surface shape of the virtual object represented by the original file 50A. Furthermore, the surface shape of the virtual object represented by the file 50C is rougher than the surface shape of the virtual object represented by the file 50B. That is, the files 50B and 50C have a lower quality (resolution) than the original file 50A by the extent that the file size is smaller than the original file size.

In addition, in this embodiment, as an example, the file generation unit 12 generates two types of compressed files (files 50B and 50C), but the file generation unit 12 generates only one type of compressed file (one specific compression rate). Alternatively, three or more types (three or more different compression ratios) of compressed files may be generated. For example, the file generation unit 12 may determine which compression ratio to generate a compressed file according to the file size of the original file 50A. For example, if the file size of the original file 50A is not very large, the file size can be made sufficiently small without increasing the compression ratio that much. In this case, the file generation unit 12 may generate only the file 50B.

The file storage unit 13 acquires and stores, for a first virtual object, a first file having a first file size and a second file having a second file size larger than the first file size. Remember.

In this embodiment, as an example, the first virtual object is a virtual object corresponding to the real object (rabbit) shown in FIG. 2. That is, in the present embodiment, the first virtual object is a virtual object generated based on imaging data of the real object. In the example of FIG. 2, file 50C corresponds to a first file having a first file size. Further, each of the files 50A and 50B corresponds to a second file having a second file size larger than the first file size. That is, in the example of FIG. 2, the file storage unit 13 stores a plurality (here, two) of second files (files 50A and 50B) having mutually different file sizes. Note that when the file generation unit 12 generates only the files 50A and 50B, the file 50B functions as the first file and the file 50A functions as the second file.

In this embodiment, the file storage unit 13 stores the files 50A, 50B (second file) and file 50C (first file) generated by the file generation unit 12 in association with the virtual object corresponding to the real object. do. That is, the file storage unit 13 stores (registers) a first file and a second file corresponding to each virtual object.

The file storage unit 13 manages a plurality of virtual objects (that is, a plurality of virtual objects each corresponding to a different real object) using, for example, unique identification information (object ID). For example, when a virtual object is operated on the web (that is, when displayed by a web browser of the user terminal 20), the file storage unit 13 stores a unique URI (Uniform Resource Identifier) for a file representing each virtual object. ) etc. may be assigned.

Furthermore, the first file and second file (in this embodiment, three files 50A, 50B, and 50C) regarding the same virtual object can be managed as follows.

In the first example, files 50A, 50B, and 50C are managed by file name. For example, the file storage unit 13 names the file 50A (original file) as "usagi.3d", the file 50B (the file to which 30% compression has been applied) as "usagi_a.3d", and the file 50C. (The file to which 60% compression has been applied) is managed (stored) as the file name "usagi_b.3d". In this example, "usagi" is character string information that uniquely identifies a virtual object (here, as an example, a virtual object corresponding to the real object (rabbit) shown in FIG. 2). ".3d" is an example of an identifier given to each file (data). “_a” and “_b” connected after “usagi” are information corresponding to the compression rate for the original file. According to the first example, it is possible to understand the virtual object and the compression rate based on the file name.

In the second example, the files 50A, 50B, and 50C are managed by a method of embedding information regarding the compression rate (the compression rate of the original file is 0) in the header portion of each file. In this case, the file storage unit 13 saves the files 50A, 50B, and 50C in different locations, and gives the files 50A, 50B, and 50C a common file name (for example, "usagi.3d"). good.

In the third example, multiple files 50A, 50B, and 50C corresponding to the same virtual object are managed by a management table. For example, multiple files 50A, 50B, and 50C corresponding to the same virtual object are associated with each other in the management table by a common key indicating the virtual object.

When it becomes necessary to send the data (file) of a certain virtual object to the user terminal 20 using any of the methods in the first to third examples above (for example, when a request to display a virtual object is received from the user terminal 20) (for example, when the virtual object is a virtual object), it becomes possible to access the first file and the second file corresponding to the virtual object from among the plurality of files stored in the file storage unit 13. Note that the file management method is not limited to the first to third examples above, and may be managed using methods other than these.

The display control unit 14 controls the display of the first virtual object on the user terminal 20. For example, when it becomes necessary to display a first virtual object on the display unit 24 of the user terminal 20, the display control unit 14 displays files (first file, second file) corresponding to the first virtual object on the user terminal 20. The display control described above is executed by controlling the processing provided to the. Specifically, the display control unit 14 executes a first process, a second process, and a third process described below. An example of processing by the display control unit 14 will be described below with reference to FIG. 3.

The first process transmits the first file (file 50C in the example of FIG. 2) to the user terminal 20 with priority over the second file (files 50A and 50B in the example of FIG. 24 to display the first virtual object based on the first file. That is, the display control unit 14 gives priority to the file 50C whose file size has been reduced over the original file and provides it to the user terminal 20. According to the above configuration, as shown in the upper part of FIG. 3, after a download request is generated from the user terminal 20 to the server 10, the first virtual object based on the file 50C is The time required to display the file in the area 24a) can be made shorter than when the original file (file 50A) is sent to the user terminal 20.

In the second process, after the first file (file 50C) is transmitted to the user terminal 20, or in parallel with the transmission of the first file to the user terminal 20, the second file (file 50A or file 50B) is transmitted to the user terminal 20. This is a process of transmitting to the terminal 20. In the second process, the display control unit 14 uses a first control pattern to start sending the second file to the user terminal 20 after sending the first file, and a first control pattern to start sending the second file to the user terminal 20 after sending the second file to the user terminal 20. The second control pattern may be executed in parallel with the transmission of the first file to.

Since the file size of the second file is larger than the file size of the first file, the transmission process of the first file is completed before the transmission process of the second file also in the second control pattern. However, in the second control pattern, the transmission process of the second file is executed in parallel with the transmission process of the first file, so the point in time when the transmission of the second file is completed in the second control pattern is the same as that in the first control pattern. This is a time before the time when the transmission of the second file is completed. That is, the second control pattern starts the transmission process of the second file before the transmission of the first file is completed, and the first control pattern starts the transmission process of the second file after the transmission of the first file is completed. The configuration is such that transmission of the second file can be completed earlier than the control pattern. On the other hand, the time point at which transmission of the first file is completed in the second control pattern is later than the time point at which transmission of the first file is completed in the first control pattern. In other words, in the second control pattern, while the second file can be transmitted earlier than in the first control pattern, the second file is also transmitted while the first file is being transmitted (i.e., communication resources are allocated between the first file and the second file). 2 files), the time required to transmit the first file is longer than that of the first control pattern.

Therefore, from the viewpoint of shortening the time until the first virtual object is displayed on the display unit 24 of the user terminal 20 as much as possible, the first control that sequentially executes the download of the first file and the download of the second file is recommended. pattern is more advantageous than the second control pattern. On the other hand, from the perspective of shortening the waiting time until the first virtual object is displayed (the waiting time experienced by the user) and providing high-quality data (second file) to the user as early as possible, the first The second control pattern, which performs parallel downloading of the file and the second file, is more advantageous than the first control pattern.

Therefore, the display control unit 14 may be configured to be able to switch between the first control pattern and the second control pattern. For example, the display control unit 14 may determine which of the first control pattern and the second control pattern to execute based on user settings. That is, the display control unit 14 may allow the user to select which of the first control pattern and the second control pattern to execute. Alternatively, the display control unit 14 may determine which of the first control pattern and the second control pattern to execute, depending on the above-mentioned terminal information (for example, communication speed). For example, if it is expected that the download of the first file will be completed in a sufficiently short time even if the first file and the second file are downloaded in parallel (for example, if the communication speed is set to a predetermined threshold above), it may be determined to execute the second control pattern. Note that even if the user terminal 20 is equipped with a configuration for high-speed parallel processing (for example, multithreading, etc.) or a configuration compatible with divided download, the first file can be downloaded in a sufficiently short time. It is expected that the download will be completed. Therefore, the display control unit 14 may determine which of the first control pattern and the second control pattern to execute, depending on the communication method installed in the user terminal 20. According to the above configuration, each of the above-described first control pattern and second control pattern can be appropriately switched according to the use case.

In the second process, the display control unit 14 selects a second file to be transmitted to the user terminal 20 from among the plurality of second files (files 50A and 50B in this embodiment) based on the terminal information. may be determined. For example, the display control unit 14 may determine the second file to be provided to the user terminal 20 according to the above-mentioned attributes of the user terminal 20 (application type, communication speed, processing performance, etc.).

For example, if the application running on the user terminal 20 cannot smoothly display the file 50A having the original quality (that is, if the file 50A has a quality that exceeds the range that can be processed by the application), or if the user terminal 20 If the user terminal 20 does not have the necessary processing performance (for example, specifications of processor, memory, etc.) to perform display processing using the file 50A, even if the file 50A is sent to the user terminal 20, In this case, display processing that takes advantage of the quality of the file 50A cannot be performed. Therefore, the display control unit 14 selects a quality level that does not exceed the maximum quality that can perform display processing on the user terminal 20, based on terminal information that includes information regarding at least one of the application type and processing performance of the user terminal 20. 2 file may be determined as the second file to be provided to the user terminal 20.

For example, in the example of FIG. 2, if the file 50A has a quality higher than the maximum quality that can be displayed on the user terminal 20, and the file 50B has a quality lower than the maximum quality. , the display control unit 14 may select the file 50B as the second file to be transmitted to the user terminal 20.

Further, even if the user terminal 20 has sufficient specifications (for example, performance of processor, memory, etc.) to perform display processing using the file 50A, if the communication speed of the user terminal 20 is slow, the file It takes a very long time to complete the download of 50A. Therefore, for example, the display control unit 14 transmits a second file that is expected to be downloaded to the user terminal 20 within a predetermined threshold time based on terminal information including information regarding the communication speed of the user terminal 20. It may also be determined as the second file to be provided. For example, in the example of FIG. 2, if the expected download time of the file 50A, which is assumed based on the communication speed of the user terminal 20, exceeds the threshold time, and the expected download time of the file 50B is less than or equal to the threshold time, the display control The unit 14 may select the file 50B as the second file to be transmitted to the user terminal 20.

According to the above configuration, when a plurality of second files (files 50A and 50B in the example of FIG. 2) having mutually different file sizes (that is, different qualities) are stored in the file storage unit 13, the user terminal Based on the attributes 20, it becomes possible to provide (transmit) a second file with appropriate quality to the user terminal 20. Note that in the example of FIG. 3, the file 50B is determined as the second file to be transmitted to the user terminal 20 through the above-described determination process.

In the third process, the first virtual object based on the first file (file 50C) is displayed on the display unit 24 (display area 24a), and the second file (here, as an example, the file 50B) is completed, the first virtual object based on the file 50C displayed on the display unit 24 is switched to the first virtual object based on the file 50B (see arrow P1 in FIG. 3). .

To switch from file 50C to file 50B, for example, an identifier specified as a display target on the user terminal 20 (here, an identifier corresponding to file 50C (for example, the above-mentioned URI, object ID, etc.)) is transferred to file 50B. This can be done by changing the identifier to the corresponding identifier. Furthermore, information such as the angle of view, position, distance (depth), etc. of the first virtual object based on the file 50C that was displayed on the display unit 24 immediately before switching may be configured to be carried over even after switching. In this case, it is possible to smoothly switch from the file 50C to the file 50B while maintaining the way the first virtual object is viewed by the user.

In addition, when the first virtual object based on the second file (file 50B) is displayed on the display unit 24 (display area 24a) (see the lower part of FIG. 3), the display control unit 14 Depending on the occurrence of a processing delay, display control may be performed to switch back the first virtual object based on the file 50B displayed on the display unit 24 to the first virtual object based on the first file (file 50C). good. For example, the display control unit 14 may determine whether or not a processing delay of the user terminal 20 has occurred based on information including information regarding the processing performance of the user terminal 20 (for example, the above-mentioned terminal information). . For example, the display control unit 14 may determine that a processing delay has occurred when the processor usage rate of the user terminal 20 is equal to or higher than a threshold value. When the display control unit 14 determines that a processing delay has occurred, the display control unit 14 allows the user to switch back the process (see arrow P2 in FIG. 3) from the second file (file 50B) to the first file (file 50C). The terminal 20 may be executed. Since the file size of the file 50C is smaller than that of the file 50B, the amount of calculation required for display processing (drawing processing) and the amount of memory used are smaller than those of the file 50B. Therefore, according to the display control described above, by executing the cutback process in a situation where a processing delay has occurred, it is possible to eliminate the processing delay.

The display control unit 14 also controls the display unit 24 (display area 24a) to display the first virtual object based on the second file (file 50B), and the display unit 24 to display a plurality of virtual objects including the first virtual object. When virtual objects are displayed, the first virtual object based on the file 50B displayed on the display section 24 is transferred to the first file (file 50C) according to the number of virtual objects displayed on the display section 24. Display control may be performed to switch back to the first virtual object based on the first virtual object. For example, the display control unit 14 acquires information indicating the number of virtual objects displayed on the display unit 24. For example, the display control unit 14 may acquire such information from the user terminal 20, or the display control unit 14 may acquire the virtual object displayed on the display unit 24 based on the history of sending virtual object files to the user terminal 20. You may also know the number of Then, for example, when the display control unit 14 determines that the number of displayed virtual objects is greater than or equal to a predetermined threshold, the display control unit 14 switches the second file (file 50B) to the first file (file 50C). You may cause the user terminal 20 to execute the return process (see arrow P2 in FIG. 3). According to the display control described above, processing delays can be avoided by executing cutback processing in situations where there is a high possibility that processing delays have occurred (or are likely to occur in the future) due to the large number of displayed virtual objects. It is possible to eliminate or avoid this problem.

Next, an example of the operation of the server 10 and user terminal 20 will be described with reference to FIG. 4. In step S1, the imaging unit 21 images a real object. In step S2, the transmitter 22 transmits the imaging data acquired in step S1 to the server 10. The transmitter 22 also transmits terminal information (in this embodiment, information including application type, communication speed, processing performance, etc., as an example) to the server 10. The imaging data and terminal information are received by the receiving unit 11 of the server 10.

In step S3, the file generation unit 12 generates a file 50A (second file), a file 50B (second file), and a file 50C (first file) based on the imaging data (see FIG. 2).

In step S4, the file storage section 13 stores the files 50A, 50B, and 50C generated by the file generation section 12.

Subsequently, for example, when the server 10 receives a request to display the first virtual object from the user terminal 20 as a trigger, the processes of steps S5 and S6 (first process) are executed. That is, in step S5, the display control unit 14 sends the first file (file 50C in the example of FIG. 3) to the user terminal 20 with priority over the second file (file 50B in the example of FIG. 3). . As a result, in step S6, the first virtual object based on the first file (file 50C) is displayed on the display unit 24 (see the upper part of FIG. 3).

Furthermore, in step S7, the display control unit 14 executes a second process. That is, the display control unit 14 transmits the second file (file 50B) to the user terminal 20 after transmitting the first file (file 50C) or in parallel with transmitting the first file. Since the file size of the second file (second file size) is larger than the file size of the first file (first file size), the second file is sent to the user terminal 20 after step S6. That is, the time point at which the reception of the second file is completed at the user terminal 20 is a time point after step S6.

Subsequently, in step S8, the display control unit 14 executes a third process. That is, the display control unit 14 displays the first virtual object based on the first file (file 50C) on the display unit 24 (display area 24a), and displays the second file (file 50B) to the user terminal 20. , the first virtual object based on the file 50C displayed on the display unit 24 is switched to the first virtual object based on the file 50B (see arrow P1 in FIG. 3). For example, the display control unit 14 transmits to the user terminal 20 together with the second file instruction information that instructs to "switch the first file to the second file after receiving the second file". can perform the switching process as described above.

Subsequently, in step S9, the display control unit 14 performs display control to switch back the first virtual object based on the file 50B displayed on the display unit 24 to the first virtual object based on the first file (file 50C). Determine whether or not. This determination process is performed periodically or irregularly based on terminal information (for example, information regarding processing performance) received from the user terminal 20, information indicating the number of virtual objects displayed on the display unit 24, etc. obtain.

For example, as described above, the display control unit 14 determines that a processing delay has occurred in the user terminal 20, or that the number of virtual objects displayed on the display unit 24 is greater than or equal to a threshold value. If so, the user terminal 20 is instructed to execute the switchback process (step S10). As a result, in step S11, a process of switching back from the second file (file 50B) to the first file (file 50C) is executed (see arrow P2 in FIG. 3). Note that after the cutback process is performed (that is, after step S11), the display control unit 14 determines the reason for the cutback (for example, that the above processing delay has occurred, that the number of displayed virtual objects is If it is determined that the problem (e.g., the current value is equal to or greater than a threshold value) has been resolved, the process of step S8 may be executed again.

In the server 10 described above, first, the first file (file 50C) having a small file size (first file size) is preferentially transmitted to the user terminal 20. With the above configuration, the waiting time until the first virtual object is displayed on the display unit 24 of the user terminal 20 can be shortened. Further, after the first file is transmitted, or in parallel with the first file transmission, the second file (file 50B) is transmitted to the user terminal 20. After the user terminal 20 receives the second file, the first virtual object based on the first file displayed on the display unit 24 is switched to the first virtual object based on the second file (arrow in FIG. 3). (See P1). With the above configuration, the first virtual object based on the second file, which has a larger file size and higher quality than the first file, can be presented to the user later, so that the quality of the user's virtual experience can be maintained.

Further, in this embodiment, the file generation unit 12 generates the first file and the second file of the virtual object corresponding to the real object based on the imaging data, and the file storage unit 13 generates the first file and the second file of the virtual object corresponding to the real object based on the imaging data. The first file and the second file are stored in association with the virtual object. According to the above configuration, when generating a virtual object corresponding to a real object, it is possible to automatically generate and store the first file and the second file corresponding to the virtual object.

Note that in the above embodiment, the display control system was configured by the server 10, but the display control system may be configured by the server 10 alone, or by the server 10 and the user terminal 20. That is, some of the functions of the server 10 described above may be executed by the user terminal 20. For example, in the embodiment described above, the display control unit 14 of the server 10 controls the display content of the display unit 24 of the user terminal 20, but the display control unit 14 may be provided in the user terminal 20 instead of the server 10. Further, some of the functions of the server 10 described above may be omitted. For example, in the embodiment described above, the server 10 (file generation unit 12) generated the virtual object file, but the virtual object file may be generated by a device different from the server 10.

The display control system of the present disclosure (server 10 in the above embodiment) has the following configuration.

[1] A display control system that controls the display of virtual objects displayed on a display unit of a user terminal,
a file storage unit that stores a first file having a first file size and a second file having a second file size larger than the first file size for a first virtual object that is one of the virtual objects; ,
a display control unit that controls display of the first virtual object on the user terminal,
The display control section includes:
a first process of displaying the first virtual object based on the first file on the display unit by transmitting the first file to the user terminal with priority over the second file;
a second process of transmitting the second file to the user terminal after transmitting the first file or in parallel with transmitting the first file;
Displayed on the display unit when the first virtual object based on the first file is displayed on the display unit and transmission of the second file to the user terminal is completed. performing a third process of switching the first virtual object based on the first file to the first virtual object based on the second file;
Display control system.

[2] Obtain imaging data obtained by imaging a real object that exists in real space, and create the first file and the second file of the virtual object corresponding to the real object based on the imaging data. further comprising a file generation unit that generates a file,
The file storage unit stores the first file and the second file generated by the file generation unit in association with the virtual object corresponding to the real object.
The display control system according to [1].

[3] When the first virtual object based on the second file is displayed on the display unit, the display control unit controls the display unit to display the first virtual object on the display unit depending on the occurrence of processing delay of the user terminal. cutting back the first virtual object based on the second file that has been updated to the first virtual object based on the first file;
The display control system according to [1] or [2].

[4] The display control unit displays the first virtual object based on the second file on the display unit, and displays the plurality of virtual objects including the first virtual object on the display unit. , the first virtual object based on the second file displayed on the display unit is changed to the first virtual object based on the first file according to the number of virtual objects displayed on the display unit. Cut back to the first virtual object,
The display control system according to any one of [1] to [3].

[5] In the second process, the display control unit includes a first control pattern for starting transmission of the second file to the user terminal after transmission of the first file, and a first control pattern for starting transmission of the second file to the user terminal after transmitting the first file. a second control pattern in which file transmission is performed in parallel with transmission of the first file; and a second control pattern configured to be switchable.
The display control system according to any one of [1] to [4].

[6] Further comprising a terminal information acquisition unit that acquires terminal information indicating attributes of the user terminal,
The file storage unit stores a plurality of second files having mutually different file sizes,
The display control unit determines the second file to be transmitted to the user terminal from among the plurality of second files, based on the terminal information.
The display control system according to any one of [1] to [5].

Furthermore, the block diagram used to explain the above embodiment shows blocks in functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using one physically or logically coupled device, or may be realized using two or more physically or logically separated devices directly or indirectly (e.g. , wired, wireless, etc.) and may be realized using a plurality of these devices. The functional block may be realized by combining software with the one device or the plurality of devices.

Functions include judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, These include, but are not limited to, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning. I can't.

For example, the server 10 in an embodiment of the present disclosure may function as a computer that performs the display control method of the present disclosure. FIG. 5 is a diagram illustrating an example of the hardware configuration of the server 10 according to an embodiment of the present disclosure. The server 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. Note that the user terminal 20 may also have the same hardware configuration as the server 10.

Note that in the following description, the word "apparatus" can be read as a circuit, a device, a unit, etc. The hardware configuration of the server 10 may include one or more of the devices shown in FIG. 5, or may not include some of the devices.

Each function in the server 10 is performed by loading predetermined software (programs) onto hardware such as the processor 1001 and the memory 1002, so that the processor 1001 performs calculations, controls communication by the communication device 1004, and controls the memory 1002 and the memory 1002. This is realized by controlling at least one of reading and writing data in the storage 1003.

The processor 1001, for example, operates an operating system to control the entire computer. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, registers, and the like.

Furthermore, the processor 1001 reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 to the memory 1002, and executes various processes in accordance with these. As the program, a program that causes a computer to execute at least part of the operations described in the above embodiments is used. For example, each functional unit of the server 10 (for example, the display control unit 14, etc.) may be realized by a control program stored in the memory 1002 and operated in the processor 1001, and other functional blocks may also be realized in the same way. Good too. Although the various processes described above have been described as being executed by one processor 1001, they may be executed by two or more processors 1001 simultaneously or sequentially. Processor 1001 may be implemented by one or more chips. Note that the program may be transmitted from a network via a telecommunications line.

The memory 1002 is a computer-readable recording medium, and includes at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. may be done. Memory 1002 may be called a register, cache, main memory, or the like. The memory 1002 can store executable programs (program codes), software modules, and the like to implement a display control method according to an embodiment of the present disclosure.

The storage 1003 is a computer-readable recording medium, such as an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, or a magneto-optical disk (for example, a compact disk, a digital versatile disk, or a Blu-ray disk). (registered trademark disk), smart card, flash memory (eg, card, stick, key drive), floppy disk, magnetic strip, etc. Storage 1003 may also be called an auxiliary storage device. The storage medium mentioned above may be, for example, a database including at least one of memory 1002 and storage 1003, a server, or other suitable medium.

The communication device 1004 is hardware (transmission/reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as a network device, network controller, network card, communication module, etc., for example.

The input device 1005 is an input device (eg, keyboard, mouse, microphone, switch, button, sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside. Note that the input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch panel).

Further, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses for each device.

The server 10 also includes hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). A part or all of each functional block may be realized by the hardware. For example, processor 1001 may be implemented using at least one of these hardwares.

Although this embodiment has been described in detail above, it is clear for those skilled in the art that this embodiment is not limited to the embodiment described in this specification. This embodiment can be implemented as modifications and changes without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is for the purpose of illustrative explanation and does not have any restrictive meaning with respect to this embodiment.

The order of the processing procedures, sequences, flowcharts, etc. of each aspect/embodiment described in this disclosure may be changed as long as there is no contradiction. For example, the methods described in this disclosure use an example order to present elements of the various steps and are not limited to the particular order presented.

The input/output information may be stored in a specific location (for example, memory) or may be managed using a management table. Information etc. to be input/output may be overwritten, updated, or additionally written. The output information etc. may be deleted. The input information etc. may be transmitted to other devices.

Judgment may be made using a value expressed by 1 bit (0 or 1), a truth value (Boolean: true or false), or a comparison of numerical values (for example, a predetermined value). (comparison with a value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or may be switched and used in accordance with execution. In addition, notification of prescribed information (for example, notification of "X") is not limited to being done explicitly, but may also be done implicitly (for example, not notifying the prescribed information). Good too.

Software includes instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name. , should be broadly construed to mean an application, software application, software package, routine, subroutine, object, executable, thread of execution, procedure, function, etc.

Additionally, software, instructions, information, etc. may be sent and received via a transmission medium. For example, if the software uses wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and/or wireless technology (infrared, microwave, etc.) to create a website, When transmitted from a server or other remote source, these wired and/or wireless technologies are included within the definition of transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may refer to voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may also be represented by a combination of

In addition, the information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or using other corresponding information. may be expressed.

The names used for the parameters mentioned above are not restrictive in any respect. Furthermore, the mathematical formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various names assigned to these various information elements are not restrictive in any respect, as the various information elements may be identified by any suitable name.

As used in this disclosure, the phrase "based on" does not mean "based solely on" unless explicitly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

As used in this disclosure, any reference to elements using the designations "first," "second," etc. does not generally limit the amount or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, reference to a first and second element does not imply that only two elements may be employed or that the first element must precede the second element in any way.

Where "include", "including" and variations thereof are used in this disclosure, these terms, like the term "comprising," are inclusive. It is intended that Furthermore, the term "or" as used in this disclosure is not intended to be exclusive or.

In this disclosure, when articles are added by translation, such as a, an, and the in English, the present disclosure may include that the nouns following these articles are plural.

In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." Note that the term may also mean that "A and B are each different from C". Terms such as "separate" and "coupled" may also be interpreted similarly to "different."

10... Server (display control system), 11... Receiving unit, 12... File generating unit, 13... File storage unit, 14... Display control unit, 20... User terminal, 24... Display unit, 24a... Display area, 50A... File (second file), 50B... file (second file), 50C... file (first file).

Claims

A display control system that controls the display of virtual objects displayed on a display unit of a user terminal, the system comprising:
a file storage unit that stores a first file having a first file size and a second file having a second file size larger than the first file size for a first virtual object that is one of the virtual objects; ,
a display control unit that controls display of the first virtual object on the user terminal,
The display control section includes:
a first process of displaying the first virtual object based on the first file on the display unit by transmitting the first file to the user terminal with priority over the second file;
a second process of transmitting the second file to the user terminal after transmitting the first file or in parallel with transmitting the first file;
Displayed on the display unit when the first virtual object based on the first file is displayed on the display unit and transmission of the second file to the user terminal is completed. performing a third process of switching the first virtual object based on the first file to the first virtual object based on the second file;
Display control system.
Obtaining imaging data obtained by imaging a real object that exists in real space, and generating the first file and the second file of the virtual object corresponding to the real object based on the imaging data. It further includes a file generation section to
The file storage unit stores the first file and the second file generated by the file generation unit in association with the virtual object corresponding to the real object.
The display control system according to claim 1.
When the first virtual object based on the second file is displayed on the display unit, the display control unit is configured to display the first virtual object on the display unit according to a processing delay occurrence status of the user terminal. cutting back the first virtual object based on the second file to the first virtual object based on the first file;
The display control system according to claim 1.
The display control unit displays the first virtual object based on the second file on the display unit, and displays the plurality of virtual objects including the first virtual object on the display unit. In this case, the first virtual object based on the second file displayed on the display unit is changed to the first virtual object based on the first file according to the number of virtual objects displayed on the display unit. cut back to object,
The display control system according to claim 1.
In the second process, the display control unit includes a first control pattern for starting transmission of the second file to the user terminal after transmission of the first file, and a first control pattern for starting transmission of the second file to the user terminal. and a second control pattern in which the second control pattern is configured to be switchable between the second control pattern and the second control pattern in which the first file is transmitted in parallel with the transmission of the first file.
The display control system according to claim 1.
further comprising a terminal information acquisition unit that acquires terminal information indicating attributes of the user terminal,
The file storage unit stores a plurality of second files having mutually different file sizes,
The display control unit determines the second file to be transmitted to the user terminal from among the plurality of second files, based on the terminal information.
The display control system according to claim 1.