JP7403581B2 - systems and devices - Google Patents

Description

The present invention relates to systems and devices for displaying virtual reality images.

Conventionally, based on footage of sports matches, the movement of objects such as balls used in matches is tracked, and virtual reality images that reproduce the movement of objects are generated and displayed on screens such as head-mounted displays. There are known techniques for displaying images. Non-Patent Document 1 discloses a technique for tracking the trajectory of a pitch in a baseball game and providing it as a virtual reality video.

Baseboy TRACKING TECHNOLOGY, [online], Qoncept, [searched on May 10, 2022], Internet <URL: https://baseboy.qar.jp/>

However, the above-mentioned conventional technology requires a user to wear a dedicated head-mounted display or goggles when viewing a stereoscopic virtual reality video, and while viewing the virtual reality video, Unable to watch real-time match footage. Therefore, for example, it is not possible to satisfy the needs of a user who wants to immediately see the pitcher's previous pitch in a virtual reality video, but at the same time cannot take his eyes off the game situation.

One aspect of the present invention has been made in view of the above problem, and aims to enable a user to view virtual reality video of a competition while grasping the progress of the competition in real time.

In order to solve the above problems, a system according to one aspect of the present invention is a system including a server and a device, the server is a system that includes an object that is involved in the competition when the target competition is in progress. a first acquisition unit that acquires movement information that specifies the movement of the player; and a generation unit that generates a virtual reality image of the competition by referring to the movement information, and the device includes a first acquisition unit that acquires movement information that specifies the movement of The device includes a second acquisition unit that acquires the virtual reality video, and a display unit that simultaneously displays the captured video and the virtual reality video on the single device.

A device according to another aspect of the present invention includes an acquisition unit that acquires a shot video of a target competition and a virtual reality video of the competition; and a display section that simultaneously displays.

A server or device control program that realizes the server or device on a computer by causing the computer to operate as each part (software element) included in the server or device according to each aspect of the present invention, and a computer-readable computer-readable program recorded thereon. Recording media also fall within the scope of the present invention.

Further, the control program may use various machine learning techniques in the process of causing the computer to operate as each of the units or in other processes. In this case, a program using machine learning techniques may run on a server or device, or may run on another device (for example, an edge computer or a cloud server).

According to one aspect of the present invention, a user can view virtual reality video of a competition while grasping the progress of the competition in real time.

It is an example of a block diagram showing the functional configuration of the system. An example of the arrangement of photographing devices for photographing measurement images is shown. An example of virtual reality video is shown. 3 is an example of a sequence diagram showing the flow of processing related to a processing example of the system. FIG. An example of a screen displayed on the display unit is shown. An example of a virtual reality image displayed on the display unit is shown. An example of a screen in a mode in which an avatar is operated and moved inside and outside of a stadium reproduced in a virtual space is shown.

Hereinafter, one embodiment of the present invention will be described in detail.

[1. Configuration example of system 1]
FIG. 1 is an example of a block diagram showing the functional configuration of a system 1 according to the present disclosure. The system 1 is a system for generating and displaying virtual reality images of a target competition, and includes a server 10 and one or more devices 20.

In the present disclosure, "competition" includes ball games and other sports, as well as various games involving movement of objects involved in the competition. Examples of ball games include baseball, softball, soccer, rugby, volleyball, basketball, tennis, table tennis, golf, bowling, and billiards. Examples of sports other than ball games include car racing, horse racing, bicycle racing, boat racing, auto racing, curling, badminton, swimming, skiing, skating, track and field, gymnastics, dance, judo, kendo, archery, and the like. Furthermore, examples of the various games mentioned above include darts, karuta, Hyakunin Isshu, and the like. Hereinafter, as one aspect, a case where the competition is a baseball game will be described as an example.

In the present disclosure, "objects" involved in a competition may include people or animals participating in the competition, in addition to balls, tools, etc. used in the competition. Further, the number of participants in the competition is not limited to a plurality of participants, and may be a single number of participants.

The server 10 is a device that functions as a server that supplies shot video and virtual reality video of the competition to the device 20, and includes a control section 11, a storage section 15, and a communication section 16.

In the present disclosure, "captured video" means a video shot of the general appearance of a competition that is used for broadcasting the competition, for example, a relay video such as a real-time TV broadcast, but this does not necessarily apply. Instead, it may be a recorded video. Further, part or all of the captured video may be used as a measurement video to be described later. On the other hand, a "virtual reality image" refers to an image in which the movement of an object, etc. is reproduced using CG (Computer Graphics) in a virtual three-dimensional space or two-dimensional plane. Note that real time in the present disclosure includes a delay of the time normally required in a series of processing (including communication) from photographing a subject to displaying an image on a device.

The control unit 11 is a control device that controls the entire server 10, and includes an acquisition unit (first acquisition unit) 12 and a generation unit 13.

The acquisition unit 12 acquires movement information that specifies the movement of objects involved in the competition as the target competition progresses. The movement information is, for example, information indicating the coordinates and speed of each object at each point in time during the competition. Note that the movement information may indicate that an object such as a ball is stationary at a certain point in time, or may be information that indicates the movement of an object such as a bat or a racket when the person is holding the object. There may be. Furthermore, the movement information is obtained by the control unit 11 acquiring one or more measurement images for tracking (three-dimensional measurement) the movement of the object from the photographing device 3, etc. not shown in FIG. The configuration may be such that the calculation is performed with reference to the video for use, or the acquisition unit 12 may be configured to acquire the movement information from another external server or the like.

For example, when the competition is a ball game, the acquisition unit 12 acquires movement information that specifies the movement of a ball used in the ball game as the ball game progresses.

FIG. 2 shows an example of the arrangement of the photographing device 3 that photographs measurement images. In many cases, a plurality of photographing devices 3 are arranged so as to surround the range in which the participants of the competition move in the stadium, as illustrated in FIG. 2 .

The generation unit 13 generates a virtual reality video of the competition with reference to the movement information acquired by the acquisition unit 12. By referring to the movement information, the generation unit 13 can generate a virtual reality image viewed from a viewpoint other than the position of the photographing device 3 that photographs the measurement image.

FIG. 3 shows an example of a virtual reality image of pitcher 5's pitch seen from the catcher's perspective. In the example of FIG. 3, the movement of the ball 6 is reproduced based on movement information, and the movement of the pitcher 5 is a predetermined movement. In other words, the virtual reality video in FIG. 3 is a video in which the ball 6 moving in accordance with the movement information is synthesized with a template video of a portion other than the ball 6. In many cases, the video serving as the template includes a video or a still image of the inside of a stadium or the like as a background. In this way, a part of the virtual reality video may be created before the acquisition unit 12 acquires movement information, and some objects in the virtual reality video may have movement information. The operation may be predefined without being used. In addition, particularly for objects whose movement is reproduced based on movement information, a configuration may be adopted in which a locus of movement is displayed, as exemplified by the sphere 6 in FIG. 3. In other words, the generation unit 13 may be configured to generate a virtual reality image in which a trajectory of an object whose movement is reproduced is drawn with reference to the movement information.

Further, a single virtual reality video may include a plurality of objects whose movements are reproduced based on movement information. For example, the generation unit 13 roughly calculates the movement of the ball 6 and the bat based on movement information that specifies the movement of the ball 6 thrown by the pitcher 5 and movement information that specifies the movement of the bat swung by the batter (not shown). A single virtual reality image reproduced at the same timing may be generated.

Further, the control unit 11 performs a process of acquiring a captured video from the imaging device 3 or an external server, and transmitting it to the device 20 via the communication unit 16.

The storage unit 15 is a storage device that stores various information, such as information that the generation unit 13 refers to when generating a virtual reality video, including information indicating the video serving as the above-mentioned template, and information on the stadium. Information indicating the shape, information indicating each image, etc. are stored.

The communication unit 16 is an interface that performs communication processing with an external device such as the device 20 based on control by the control unit 11. For example, the communication unit 16 performs a process of transmitting captured video and virtual reality video to the device 20.

The device 20 is a device that displays the received captured images and virtual reality images on a screen, and is, for example, a mobile communication terminal such as a smartphone or a tablet. The device 20 includes a control section 21, a storage section 25, a communication section 26, a display section 27, and an input section 28.

The control unit 21 is a control device that controls the entire device 20, and includes an acquisition unit (second acquisition unit) 22. The acquisition unit 22 acquires the shot video of the competition and the virtual reality video transmitted from the server 10.

The storage unit 25 is a storage device that stores various information, and stores information indicating display settings on the device 20, information for specifying the server 10 as a communication destination, and the like.

The communication unit 26 is an interface that performs communication processing with an external device such as the server 10 under the control of the control unit 21 . For example, the communication unit 26 receives captured video and virtual reality video from the server 10.

The display unit 27 is a display that displays still images, videos, etc. on a screen based on the control by the control unit 21. The input unit 28 is an interface such as a button that receives user instructions for the device 20. Note that the display section 27 and the input section 28 may be realized as an integrated touch pad, or the input section 28 may be a remote control or controller of a separate device from the main body of the device 20. You can.

Note that the function of a single member included in the system 1 may be realized by another plurality of members, and the functions of the plurality of members included in the system 1 may be realized by another single member. For example, the server 10 may have a configuration realized by a first server having a database and a second server communicating with the device 20.

[2. Processing example of system 1]
Next, the flow of processing executed by the system 1 will be described using an example. FIG. 4 is an example of a sequence diagram showing the flow of processing according to this example.

In S101 (step S101), the control unit 11 of the server 10 acquires a captured video of the target competition and a measurement video from the above-mentioned imaging device 3 or the like via the communication unit 16.

In S102, the control unit 11 calculates movement information with reference to the measurement video. Further, the acquisition unit 12 acquires the movement information.

In S103, the generation unit 13 generates a virtual reality video of the competition with reference to the movement information acquired by the acquisition unit 12. Information indicating each video image acquired or generated by the control unit 11 in S101 to S103 is at least temporarily stored in the storage unit 15.

In S104, the control unit 21 of the device 20 transmits information requesting transmission of the captured video and virtual reality video to the server 10 via the communication unit 26. The process of S104 is executed when the input unit 28 receives a user's instruction to start displaying the photographed video and the virtual reality video.

In S105, the control unit 11 of the server 10 transmits the captured video and the virtual reality video to the requesting device 20.

In S106, the control unit 21 of the device 20 arranges the photographed video and the virtual reality video received from the server 10 and acquired by the acquisition unit 22 at specified positions, and displays them simultaneously on the display unit 27.

Note that the order in which the processes in FIG. 4 are executed is not limited to the above-described order; for example, the server 10 may be configured to generate a virtual reality video in parallel with acquiring the captured video, The process in S104 may be executed before or after S101 or S102.

Furthermore, the timing at which the device 20 requests transmission of the photographed video and the timing at which the virtual reality video is requested to be transmitted may not be the same, and the timing at which the server 10 starts transmitting the photographed video and the transmission of the virtual reality video may not be the same. It does not have to be at the same time as the start timing. Alternatively, the virtual reality video may be configured to be transmitted to the device 20 that is displaying the shot video as soon as it is generated, regardless of a request from the device 20.

Note that the server 10 may accept a request to transmit each of the captured images or virtual reality images before acquiring or generating the images. In the above case, the server 10 transmits each video to the device 20 as soon as transmission becomes possible.

According to the configuration of this example, the user can view the virtual reality video of the competition while grasping the progress of the competition in real time.

[3. Display screen example]
Next, an example of a screen displayed on the display unit 27 of the device 20 will be described. The control unit 21 generates a screen according to the user's instruction input to the input unit 28 and causes the display unit 27 to display the screen.

Screen 31 and screen 36 in FIG. 5 show examples of screens displayed on a single display, which is display unit 27, in the process of S106 described above.

The “smartphone mode” shown on the screen 31 is a mode in which the user views the device 20 while holding it in his hand or placing it on a stand on his desk. On the screen 31, a captured video 33 is displayed superimposed on a virtual reality video 34.

Note that the arrangement of each video may be reversed, and the virtual reality video 34 may be superimposed on the captured video 33 displayed on the entire screen. That is, the display unit 27 may display one of the photographed video and the virtual reality video superimposed on the other. This makes it possible to improve the visibility of each video. Therefore, even when the user is watching the virtual reality video 34, for example, the captured video 33 comes into the user's field of vision at the same time, making it easier for the user to grasp the progress of the competition in real time.

In one aspect, the shot video 33 is a real-time relay video, and the virtual reality video 34 is a virtual reality video that reproduces a play such as pitching corresponding to the relay video immediately after. Further, the same explanation applies to the photographed video 38A (38B) and the virtual reality video 39A (39B). In the above embodiment, for example, the play in the relayed shot video 33 can be confirmed in the virtual reality video 34 immediately after.

Since it takes time to generate the virtual reality video 34, there is usually a time difference of about 0.5 seconds between the display timing of each play between the captured video 33 and the virtual reality video 34. Accordingly, the display timings may be matched by delaying the reproduction of the photographed video 33 by the time difference. Further, the display unit 27 may further increase the time difference and display the screen according to a user's instruction.

“VR mode (Virtual Reality mode)” shown on the screen 36 is a mode in which the smartphone, which is the device 20, is set in a wearable device such as dedicated goggles for viewing. That is, the VR mode is a mode in which viewing is performed in a state where the device 20 and the wearable device constitute a single device. Note that the device 20 may be considered to include the wearable device.

The virtual reality video 39A and the virtual reality video 39B included in the screen 36 are videos of the same content with mutual parallax, and the user can view the virtual reality videos 39A and 39B as a stereoscopic video with depth. . That is, when viewing the screen 36 using special goggles or the like, the pitcher's pitch appears to be approaching the user.

Note that in many cases, the photographed images 38A and 38B are two-dimensional images with no parallax, but this is not necessarily the case.

The display unit 27 switches between displaying a "smartphone mode" screen 31 and a "VR mode" screen 36 in response to a user's instruction. Further, the display unit 27 may display only the photographed video on the screen, or may display only the virtual reality video, on the screen, depending on the user's instructions. Moreover, the control unit 21 does not need to acquire images that are not to be displayed on the display unit 27 from the server 10.

FIG. 6 is another example of the virtual reality image displayed on the display unit 27, and shows an example of the virtual reality image viewed from the viewpoint of the umpire located behind the catcher. No photographed video is displayed on the screen in FIG.

As illustrated in FIG. 6, the virtual reality image includes the score 40 of the competition, the names of the participants in the competition, 41 and 42, the name of the next batter 43, the position 44 of other participants, and the current Information such as the score 45 during the competition may be included on the screen. In other words, the generation unit 13 may be configured to generate a virtual reality video including the information separately acquired by the control unit 11.

Further, FIG. 7 shows a mode different from the mode illustrated in FIG. 6 etc. in which images of the competition are displayed, in which the avatar 51 is operated to move inside and outside the stadium 50 reproduced in the virtual space. An example of a screen in this mode is shown.

In FIG. 7, button 53 is a button for moving avatar 51. In FIG. Buttons 54 and 55 are buttons for causing the avatar to perform specific actions. For example, an action such as a jump is assigned to the button 55. The button 57 is a button for chatting with another user who operates a separate device 20. Note that the above-mentioned virtual space may be a location shared with other users. That is, in the virtual space, another avatar 51 operated by another user may be displayed, and the user may be able to have a conversation with the avatar 51 operated by the user. The button 58 is a button for displaying a map of the virtual space. The button 59 is a button for performing various settings such as the appearance of the avatar.

The display unit 27 switches between a mode for displaying competition images and a mode for moving the avatar 51 according to a user's instruction. Further, when the avatar 51 is moved to a predetermined position such as the audience seats in the stadium, the mode may be such that the mode automatically changes to display the video of the competition. According to the configuration illustrated in FIG. 7, in addition to improving the game quality of video display using the device 20, it also improves the sense of immersion and contributes to making the user feel like they are actually at the stadium. .

[4. Modified example]
The captured video of the competition is not limited to the configuration in which the server 10 supplies the device 20. In other words, the source from which the device 20 acquires the shot video may be an external server or the like that is different from the server 10 that supplies the virtual reality video.

Further, the process of arranging the photographed video and the virtual reality video on the screen, for example, the process of superimposing one of the photographed video and the virtual reality video on the other, is performed not by the device 20 but by the control unit 11 of the server 10. The configuration may be such that it is performed.

Further, in relation to FIG. 6, the process of arranging information such as the competition score 40 in the virtual reality video may be performed by the control unit 21 of the device 20 by acquiring the information from the server 10 or the like. good.

Alternatively, the device 20 may include a plurality of displays as the display section 27, and the first display may display a captured video and the second display may display a virtual reality video. Furthermore, in relation to the screen 36 in FIG. 5, the first display may display the photographed image 38A and the virtual reality image 39A, and the second display may display the photographed image 38B and the virtual reality image 39B. good.

Further, the display unit 27 may display the photographed video and the virtual reality video in the same size, or may display them in different sizes. In addition, the display unit 27 may display at least one of the photographed video and the virtual reality video in a size according to the user's instruction. Furthermore, the arrangement of each video is not limited to a specific arrangement, and may be configured to be changeable according to user instructions.

Further, the generation unit 13 of the server 10 generates a plurality of virtual reality images, and the display unit 27 of the device 20 displays one or more virtual reality images specified by the user among the plurality of virtual reality images in an arbitrary manner. The configuration may be such that the information is displayed at specific timings. The above-mentioned configuration is realized by the control unit 11 of the server 10 transmitting a virtual reality video designated by the user from among the virtual reality videos accumulated in the storage unit 15 to the device 20. Further, the plurality of virtual reality videos may be videos obtained by dividing a video of one game of a certain competition into a plurality of videos. This allows the user to view, for example, virtual reality images corresponding to a specific batter, virtual reality images corresponding to the first pitches thrown to each batter, etc. all at once.

Further, the plurality of virtual reality videos described above generated by the generation unit 13 may be virtual reality videos from different viewpoints in the same scene of the competition. Further, the display unit 27 may display a plurality of virtual reality images simultaneously. For example, the display unit 27 may simultaneously display a captured video of the entire baseball game, a virtual reality video seen from the perspective of the catcher, and a virtual reality video seen from the viewpoint of the batter, runner, manager, or umpire.

Furthermore, the display unit 27 may be configured to display a virtual reality image only when an object involved in the competition makes a predetermined movement, that is, when the movement of the object satisfies a predetermined condition. In one embodiment, the generation unit 13 may be configured not to generate the virtual reality video if the movement information does not indicate that an object involved in the competition has made a predetermined movement. For example, the display unit 27 constantly displays only the shot video out of the shot video and the virtual reality video, and displays the virtual reality video that reproduces the pitch only when the ball speed of the pitch exceeds a predetermined speed. It may also be displayed in a superimposed manner.

[5. Example of implementation using software]
The functions of the server 10 and the device 20 (hereinafter referred to as "apparatus") are programs for making a computer function as the apparatus, and each control block of the apparatus (particularly included in the control unit 11 or 21) It can be realized by a program for making a computer function as each part).

In this case, the device includes a computer having at least one control device (for example, a processor) and at least one storage device (for example, a memory) as hardware for executing the program. By executing the above program using this control device and storage device, each function described in each of the above embodiments is realized.

The above program may be recorded on one or more computer-readable recording media instead of temporary. This recording medium may or may not be included in the above device. In the latter case, the program may be supplied to the device via any transmission medium, wired or wireless.

Further, part or all of the functions of each of the control blocks described above can also be realized by a logic circuit. For example, an integrated circuit in which a logic circuit functioning as each of the control blocks described above is formed is also included in the scope of the present invention. In addition to this, it is also possible to realize the functions of each of the control blocks described above using, for example, a quantum computer.

Further, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI may operate on the control device, or may operate on another device (for example, an edge computer or a cloud server).

The present invention is not limited to the embodiments described above, and various modifications can be made within the scope of the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. are also included within the technical scope of the present invention.

1 System 10 Servers 11, 21 Control unit 12 Acquisition unit (first acquisition unit)
13 Generation unit 15, 25 Storage unit 16, 26 Communication unit 20 Device 22 Acquisition unit (second acquisition unit)
27 Display section 28 Input section

Claims (3)

A system comprising a server and a device,
The server is
a first acquisition unit that acquires movement information that specifies the movement of objects involved in the competition as the target competition progresses;
a generation unit that generates a first virtual reality image of the competition with reference to the movement information,
The device includes:
a second acquisition unit that acquires the photographed video of the competition and the first virtual reality video;
a reception unit that accepts a user's operation to move the avatar in and out of a stadium reproduced in a virtual space different from the first virtual reality image;
A first mode in which the captured video and the first virtual reality video are simultaneously displayed in the single device, and a display that corresponds to the movement of an avatar that moves in response to a user's operation received by the reception unit. a display unit having a second mode;
Equipped with
The display section is
When a user moves the avatar to a predetermined position of a stadium reproduced in the virtual space, automatically switching the display from the second mode to the first mode;
In the first mode, after displaying the movement of an object involved in the competition in the photographed video, the movement of the object involved in the competition displayed in the photographed video is reproduced in the first virtual reality video. The system to display as such. A device,
an acquisition unit that acquires a captured video of a target competition and a first virtual reality video of the competition;
a display unit that simultaneously displays the captured video and the first virtual reality video in the single device;
a reception unit that accepts a user's operation to move the avatar in and out of a stadium reproduced in a virtual space different from the first virtual reality image;
Equipped with
The display section is
After displaying the movement of an object involved in the competition in the photographed video, displaying in the first virtual reality video so as to reproduce the movement of the object involved in the competition displayed in the photographed video,
A first mode in which the captured video and the first virtual reality video are simultaneously displayed on the single device, and a second mode in which an avatar that moves in response to a user's operation received by the reception unit is displayed. Equipped with
A device that automatically switches and displays the second mode to the first mode when a user moves the avatar to a predetermined position of a stadium reproduced in the virtual space. The display unit displays, in the captured video, the progress of the competition in the future from the movement of the object involved in the competition at the timing when the movement of the object involved in the competition is displayed in the first virtual reality video. 3. The device of claim 2 .

Images