JP2023163133A - Image processing system, image processing method, and computer program - Google Patents

Abstract

To provide an image processing system that displays an attractive digital content including a virtual viewpoint image and other images.SOLUTION: A first region 911 and a second region 912 exist in a graphic user interface (GUI) image that is shown on a user device. A first display region 901 to a sixth display region 906 that are of a three-dimensional shape, and that display images that show information associated with the digital content are included in the first region. Respectively allocated images and information are shown in each display region. A seventh display region 907 exists in the second region 912, that shows an image indicative of information associated with the display region selected by a user among the first to the sixth display regions. When the second display region 902 is selected, to which three virtual viewpoint images are correlated, a GUI 908 to a GUI 910 corresponding to the virtual viewpoint image of each viewpoint are displayed in the second region. The GUI 908 to GUI 910 may be displayed by superimposition in the seventh display region 907.SELECTED DRAWING: Figure 9

Description

The present disclosure relates to an image processing system, an image processing method, a computer program, and the like.

A technique that generates a virtual viewpoint image from a specified virtual viewpoint using a plurality of images obtained by imaging by a plurality of imaging devices is attracting attention. Patent Document 1 describes a method of capturing images of a subject by installing a plurality of imaging devices at different positions, and generating a virtual viewpoint image using a three-dimensional shape of the subject estimated from the captured image obtained by the capturing. Are listed.

JP2015-45920A

However, it has not been possible to provide attractive digital content including virtual perspective images and other images.

The present disclosure is directed to providing an image processing system for providing attractive digital content display techniques including virtual perspective images and other images.

An image processing system according to one embodiment of the present disclosure includes:
A virtual viewpoint image associated with the first surface of three-dimensional digital content, which is generated based on a plurality of images obtained by capturing images with a plurality of imaging devices and a virtual viewpoint. and identifying means for identifying an image of a different viewpoint from the virtual viewpoint corresponding to the virtual viewpoint image associated with the second side of the digital content;
Display control means for controlling display of an image corresponding to the virtual viewpoint image and an image corresponding to an image from a viewpoint different from the virtual viewpoint in a display area;
It is characterized by having the following.

According to the present disclosure, attractive digital content including virtual perspective images and other images can be displayed.

1 is a diagram illustrating an example of a device configuration of an image processing system 100 according to a first embodiment. FIG. 1 is a diagram showing a hardware configuration of an image processing system 100 according to a first embodiment. 3 is a flowchart for explaining the operation flow of the image processing system 100 according to the first embodiment. FIGS. 4A to 4C are diagrams showing examples of stereoscopic images as content generated by the content generation unit 4 in the fourth embodiment. 12 is a flowchart for explaining the operation flow of the image processing system 100 according to the fifth embodiment. 7 is a flowchart for explaining the operation flow of the image processing system 100 according to the third embodiment. 7 is a flowchart continuing from FIG. 6. 7 is a flowchart that is a continuation of FIGS. 6 and 7. FIG. FIG. 12 is a diagram illustrating an example of a graphical user interface displayed on a user device according to a fourth embodiment. 12 is a flowchart for explaining the operation flow of Embodiment 4. 12 is a diagram illustrating an example of a graphical user interface displayed on a user device according to Embodiment 5. FIG. 13 is a flowchart for explaining the operation flow of the fifth embodiment. FIG. 12 is a diagram illustrating an example of a graphical user interface displayed on a user device according to a sixth embodiment. 13 is a flowchart for explaining the operation flow of the sixth embodiment. FIG. 7 is a diagram illustrating each side of three-dimensional digital content according to a seventh embodiment. FIG. 7 is a diagram illustrating a shooting direction for a player according to a seventh embodiment. 12 is a diagram showing an example of three-dimensional digital content generated by the content generation unit 4 in Embodiment 7. FIG. 10 is a diagram showing an example of the device configuration of an image processing system 101 according to a seventh embodiment. FIG. 12 is a flowchart for explaining the operation flow of the image processing system 101 according to the seventh embodiment. 12 is a flowchart for explaining the operation flow of the image processing system 101 according to the eighth embodiment. 10 is a diagram showing a system configuration of an image processing system 103 according to a ninth embodiment. FIG. FIG. 7 is a diagram showing the flow of data transmission in Embodiment 9. FIG.

Embodiments of the present disclosure will be described below with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In each figure, the same reference numerals are given to the same members or elements, and overlapping explanations are omitted or simplified.

(Embodiment 1)
The image processing system of Embodiment 1 generates a virtual viewpoint as seen from the virtual viewpoint based on captured images obtained by imaging from different directions with a plurality of imaging devices (cameras), the state of the imaging device, and a specified virtual viewpoint. Generate an image. Then, the virtual viewpoint image is displayed on the surface of the virtual stereoscopic image. Note that the imaging device may include not only a camera but also a functional unit that performs image processing. Furthermore, the imaging device may include a sensor that acquires distance information in addition to the camera.

The multiple cameras capture images of the imaging area from multiple directions. The imaging area is, for example, an area surrounded by the field of a stadium at an arbitrary height. The imaging area may correspond to the three-dimensional space in which the three-dimensional shape of the subject described above is estimated. The three-dimensional space may be the entire imaging region or a part thereof. Furthermore, the imaging area may be a concert venue, an imaging studio, or the like.

The plurality of cameras are installed at different positions and different directions (attitudes) so as to surround the imaging area, and perform imaging in synchronization. Note that the plurality of cameras do not have to be installed all around the imaging area, and may be installed only in a part of the imaging area depending on restrictions on the installation location. The number of cameras is not limited; for example, if the imaging area is a rugby stadium, tens to hundreds of cameras may be installed around the stadium.

Furthermore, the plurality of cameras may include cameras with different angles of view, such as a telephoto camera and a wide-angle camera. For example, by capturing an image of a player at high resolution using a telephoto camera, the resolution of the generated virtual viewpoint image can be improved. Furthermore, in the case of a ball game, the movement range of the ball is wide, so by capturing images using a wide-angle camera, the number of cameras can be reduced. Further, by capturing images by combining the imaging areas of the wide-angle camera and the telephoto camera, the degree of freedom in the installation position is improved. Note that the cameras are synchronized at a common time, and imaging time information is added to each frame of the captured image.

A virtual viewpoint image, also called a free viewpoint image, allows an operator to monitor an image corresponding to a freely (arbitrarily) specified viewpoint. Included in the virtual viewpoint image is the one that monitors the image corresponding to the virtual viewpoint image. Further, the virtual viewpoint may be specified by an operator operation, or may be automatically specified by AI based on the results of image analysis. Further, the virtual viewpoint image may be a video or a still image.

The virtual viewpoint information used to generate the virtual viewpoint image is information including the position and direction (posture) of the virtual viewpoint, as well as the angle of view (focal length). Specifically, the virtual viewpoint information includes a parameter representing the three-dimensional position of the virtual viewpoint, a parameter representing the direction (line-of-sight direction) from the virtual viewpoint in pan, tilt, and roll directions, focal length information, and the like. However, the content of the virtual viewpoint information is not limited to the above.

Further, the virtual viewpoint information may include parameters for each of a plurality of frames. That is, the virtual viewpoint information may have parameters corresponding to each of a plurality of frames constituting the video of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive points in time.

The virtual viewpoint image is generated, for example, by the following method. First, images from multiple cameras are acquired by capturing images from different directions using cameras. Next, a foreground image in which a foreground region corresponding to a subject such as a person or a ball is extracted, and a background image in which a background region other than the foreground region is extracted are obtained from the multiple camera images. The foreground image and the background image have texture information (color information, etc.).

Then, a foreground model representing the three-dimensional shape of the subject and texture data for coloring the foreground model are generated based on the foreground image. Furthermore, texture data for coloring a background model representing the three-dimensional shape of a background such as a stadium is generated based on the background image. Then, a virtual viewpoint image is generated by mapping texture data to the foreground model and background model and performing rendering according to the virtual viewpoint indicated by the virtual viewpoint information.

However, the method for generating a virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of a captured image without using a foreground or background model.

The foreground image is an image in which a subject area (foreground area) is extracted from a captured image captured and acquired by a camera. The subject extracted as a foreground region refers to a dynamic subject (moving object) that moves (its absolute position and shape may change) when images are captured from the same direction in time series. The subject includes, for example, people such as players and referees on the field in a competition, and a ball in addition to people in the case of a ball game. Furthermore, in concerts and entertainment, singers, performers, performers, presenters, and the like are subjects in the foreground.

The background image is an image of an area (background area) that is different from at least the subject that is the foreground. Specifically, the background image is an image obtained by removing the foreground subject from the captured image. Furthermore, the background refers to an object to be imaged that remains stationary or nearly stationary when images are taken from the same direction over time.

Such imaging targets include, for example, stages for concerts, stadiums for events such as competitions, structures such as goals used in ball games, fields, and the like. However, the background is at least a different area from the subject that is the foreground. Note that the imaging target may include other objects in addition to the subject and the background.

FIG. 1 is a diagram showing an image processing system 100 of this embodiment. Note that some of the functional blocks shown in FIG. 1 are realized by causing a computer included in the image processing system 100 to execute a computer program stored in a memory as a storage medium. However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), etc. can be used.

Furthermore, the respective functional blocks of the image processing system 100 do not need to be built into the same housing, and may be configured by separate devices connected to each other via signal paths. Image processing system 100 is connected to multiple cameras. The image processing system 100 also includes a shape estimation section 2, an image generation section 3, a content generation section 4, a storage section 5, a display section 115, an operation section 116, and the like. The shape estimation section 2 is connected to the plurality of cameras 1 and the image generation section 3 , and the display section 115 is connected to the image generation section 3 . Note that each functional block may be implemented in separate devices, or all or some of the functional blocks may be implemented in the same device.

The plurality of cameras 1 are arranged at different positions around a stage for a concert or the like, a stadium for an event such as a competition, a structure such as a goal used in a ball game, a field, etc., and take images from different viewpoints. Furthermore, each camera has an identification number (camera number) for identifying the camera. The camera 1 may also include other functions, such as a function of extracting a foreground image from a captured image, and hardware (circuits, devices, etc.) for realizing the functions. The camera number may be set based on the installation position of the camera 1, or may be set based on other criteria.

The image processing system 100 may be placed inside the venue where the camera 1 is installed, or may be placed outside the venue, for example at a broadcasting station. The image processing system 100 is connected to the camera 1 via a network.

The shape estimation unit 2 acquires images from a plurality of cameras 1. Then, the shape estimation unit 2 estimates the three-dimensional shape of the subject based on the images acquired from the plurality of cameras 1. Specifically, the shape estimation unit 2 generates three-dimensional shape data expressed using a known expression method. The three-dimensional shape data may be point cloud data made up of points, mesh data made up of polygons, or voxel data made up of voxels.

The image generation unit 3 acquires information indicating the position and orientation of the three-dimensional shape data of the subject from the shape estimation unit 2, and calculates the two-dimensional shape of the subject expressed when the three-dimensional shape of the subject is viewed from a virtual viewpoint. It is possible to generate a virtual viewpoint image including: In addition, in order to generate a virtual viewpoint image, the image generation unit 3 receives designation of virtual viewpoint information (position of virtual viewpoint, line of sight direction from the virtual viewpoint, etc.) from the operator, and generates a virtual viewpoint based on the virtual viewpoint information. Images can also be generated. Here, the image generation unit 3 functions as a virtual viewpoint image generation means that generates a virtual viewpoint image based on a plurality of images obtained from a plurality of cameras.

The virtual viewpoint image is sent to the content generation unit 4, and the content generation unit 4 generates, for example, three-dimensional digital content as described later. Further, the digital content generated by the content generation unit 4 and including the virtual viewpoint image is output to the display unit 115.

Note that the content generation section 4 can also directly receive images from a plurality of cameras and supply images from each camera to the display section 115. Furthermore, based on instructions from the operation unit 116, it is also possible to switch on which side of the virtual stereoscopic image the image for each camera and the virtual viewpoint image are to be displayed.

The display unit 115 is composed of, for example, a liquid crystal display, an LED, or the like, and acquires digital content including a virtual viewpoint image from the content generation unit 4 and displays it. Furthermore, a GUI (Graphical User Interface) for an operator to operate each camera 1 is displayed.

Further, the operation unit 116 includes a joystick, a jog dial, a touch panel, a keyboard, a mouse, and the like, and is used by an operator to operate the camera 1 and the like. Further, the operation unit 116 is used by the operator to select an image to be displayed on the surface of the digital content (stereoscopic image) generated by the content generation unit 4. Furthermore, the position and orientation of a virtual viewpoint for generating a virtual viewpoint image in the image generation unit 3 can be specified.

Note that the position and orientation of the virtual viewpoint may be directly specified on the screen by an operator's operation instruction. Alternatively, when a predetermined subject is specified on the screen by an operator's operation instructions, the predetermined subject is image recognized and tracked, and virtual viewpoint information from the subject and an arc-shaped surrounding area centered on the subject are displayed. The virtual viewpoint information from the position may be automatically specified.

Furthermore, the system automatically recognizes objects that meet pre-specified conditions based on operator instructions, and automatically specifies virtual viewpoint information from that object and virtual viewpoint information from positions in an arc around the object. You may also do this. In this case, the specified conditions include, for example, the name of a specific athlete, the person who made the shot, the person who made a fine play, the position of the ball, and the like.

The storage unit 5 includes a memory for storing digital content generated by the content generation unit 4, virtual viewpoint images, camera images, and the like. Further, the storage unit 5 may include a removable recording medium. The removable recording medium can store, for example, multiple camera images taken at other venues or other sports scenes, virtual viewpoint images generated using them, digital content generated by combining them, etc. It may be recorded.

Furthermore, the storage unit 5 is capable of storing a plurality of camera images downloaded from an external server etc. via a network, virtual viewpoint images generated using them, digital content generated by combining them, etc. It's okay. Furthermore, these camera images, virtual viewpoint images, digital content, etc. may be created by a third party.

FIG. 2 is a diagram showing the hardware configuration of the image processing system 100 according to the first embodiment, and the hardware configuration of the image processing system 100 will be described using FIG. 2.

The image processing system 100 includes a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, a display section 115, an operation section 116, a communication I/F 117, a bus 118, and the like. The CPU 111 implements each functional block of the image processing system shown in FIG. 1 by controlling the entire image processing system 100 using computer programs stored in the ROM 112, RAM 113, auxiliary storage device 114, and the like.

The RAM 113 temporarily stores computer programs and data supplied from the auxiliary storage device 114, data supplied from the outside via the communication I/F 117, and the like. The auxiliary storage device 114 is composed of, for example, a hard disk drive, and stores various data such as image data, audio data, and digital content including virtual viewpoint images from the content generation unit 4.

As described above, the display unit 115 displays digital content including virtual viewpoint images, a GUI, and the like. As described above, the operation unit 116 receives operation input from the operator and inputs various instructions to the CPU 111. The CPU 111 operates as a display control unit that controls the display unit 115 and an operation control unit that controls the operation unit 116.

The communication I/F 117 is used for communication with devices external to the image processing system 100 (for example, the camera 1, an external server, etc.). For example, when the image processing system 100 is connected to an external device by wire, a communication cable is connected to the communication I/F 117. When the image processing system 100 has a function of wirelessly communicating with an external device, the communication I/F 117 includes an antenna. Bus 118 connects each part of image processing system 100 to transmit information.

Note that although the present embodiment shows an example in which the display unit 115 and the operation unit 116 are included inside the image processing system 100, at least one of the display unit 115 and the operation unit 116 is included in the image processing system 100. It may exist as a separate device externally. Note that the image processing system 100 may be in the form of, for example, a PC terminal.

FIG. 3 is a flowchart for explaining the operation flow of the image processing system 100 of the first embodiment. Further, FIGS. 4A to 4C are diagrams showing examples of three-dimensional digital content generated by the content generation unit 4 in the first embodiment.

Note that each step in the flowchart of FIG. 3 is performed by the CPU 111 as a computer of the image processing system 100 executing a computer program stored in a memory such as the ROM 112 or the auxiliary storage device 114.

In this embodiment, the image processing system 100 may be installed at a broadcasting station or the like to produce and broadcast cubic-shaped digital content 200 as shown in FIG. 4(A), or may be provided via the Internet. You may. At that time, NFT can be added to the digital content 200.

That is, in order to improve asset value, for example, the quantity of content to be distributed is limited and managed by serial numbers, thereby making the content rare. Note that NFT is an abbreviation for Non-fungible Token, and is a token to be issued and distributed on a blockchain. Examples of NFT formats include token standards called ERC-721 and ERC-1155. Tokens are typically stored in association with a wallet controlled by an operator.

In step S31, the CPU 111 associates the main camera image (first image) with the first surface 201 of the digital content 200 having a three-dimensional shape as shown in FIG. 4(A), for example. Note that the main camera image associated with the first surface 201 may be displayed for operator confirmation. Furthermore, as shown in FIG. 4(A), the direction of the line of sight from a virtual viewpoint of the digital content (specifically, the direction perpendicular to the page of FIG. 4(A)) and the direction of the first surface 201 If the normal direction is not parallel, the following display may be performed. That is, the main camera image displayed on the first surface 201 may be generated by projective transformation depending on the angle of the normal direction of the first surface 201 with respect to the display surface of the digital content. Here, the main camera image (main image, first image) is an image selected for TV broadcasting etc. from among multiple images obtained from multiple cameras installed at a sports venue. . Note that the main image is an image that includes a predetermined subject within the angle of view. Furthermore, the main camera image does not have to be captured by a camera installed at a sports venue. For example, the image may be an image captured by a camera carried by a photographer. Alternatively, it may be an image taken by a camera brought by a spectator in the venue or an electronic device such as a smartphone equipped with a camera. Further, the main camera image may be one of the plurality of cameras used to generate the virtual viewpoint image, or may be a camera not included in the plurality of cameras.

An operator of a broadcasting station or the like uses the operation unit 116 to sequentially select which camera's image is to be broadcast or distributed over the Internet as the main image. For example, when broadcasting or distributing the moment of a goal, an image from a camera near the goal is often broadcast as the main image.

In this embodiment, as shown in FIGS. 4A to 4C, the surface visible on the left side is the first surface, the surface visible on the right side is the second surface, and the surface visible on the upper side is the third surface. It is a face. However, it is not limited to this. It is assumed that which surfaces are to be designated as the first to third surfaces can be arbitrarily set in advance.

In step S32, the content generation unit 4 associates the third page 203 of the digital content 200 with data such as the name of the player who shot on goal, the name of the team he belongs to, and the final game result of the game in which he scored, as accompanying data. Incidentally, the associated data may be displayed on the third page 203 for confirmation by the operator. When providing NFTs, data representing rarity, such as the number of NFTs issued, may be displayed on the third page 203 as accompanying data. The number of issues may be determined by an operator who uses the image generation system to generate digital content, or may be automatically determined by the image generation system.

In step S33, the image generation unit 3 generates an image that is different from the viewpoint direction of the camera that captures the main camera image by a predetermined angle (for example, 90 degrees) and that includes, for example, a goal or a shooter, from a plurality of cameras. Obtain from among the images from 1. At this time, since the placement positions and postures of the multiple cameras are known in advance, the CPU 111 can determine from which camera images with different predetermined angular viewpoints as described above can be obtained from the main camera image. be. Note that although the expression "image viewpoint" is sometimes used below, it refers to the viewpoint of a camera that captures an image or a virtual viewpoint designated for generating an image.

Alternatively, in step S33, the image generation unit 3 acquires a virtual viewpoint image from a predetermined virtual viewpoint (for example, the orientation of the viewpoint is different by 90 degrees as described above) and includes the image-recognized subject. It's okay. In that case, the virtual viewpoint may be acquired by specifying the above-mentioned predetermined virtual viewpoint (with a different 90-degree viewpoint direction, ie, posture, as described above) to the image generation unit 3, and causing the image generation unit to generate a virtual viewpoint image.

Alternatively, virtual viewpoint images of a plurality of viewpoints may be generated in advance by the image generation unit 3, and a corresponding one may be selected from among the virtual viewpoint images to obtain the virtual viewpoint images. Note that in this embodiment, an image whose viewpoint differs from the main camera image by a predetermined angle is, for example, an image whose viewpoint differs by 90 degrees, but this angle can be set in advance.

Further, the virtual viewpoint image may be an image corresponding to a virtual viewpoint specified based on the orientation of the subject included in the main camera image (for example, the orientation of the face or body in the case of a person). Note that when there are multiple subjects included in the main camera image, a virtual viewpoint may be set for one of the subjects, or a virtual viewpoint may be set for a plurality of subjects.

Note that the example in which a viewpoint at a predetermined angle is selected with respect to the main image has been described above. However, for example, a virtual viewpoint image from one predetermined viewpoint is selected and acquired from one of the subject's viewpoint, a viewpoint from behind the subject, and a virtual viewpoint located on an arc centered on the subject. You can do it like this.

The subject viewpoint is a virtual viewpoint in which the position of the subject is the position of the virtual viewpoint, and the orientation of the subject is the line of sight direction from the virtual viewpoint. For example, when the subject is a person, the subject viewpoint is a viewpoint in which the position of the person's face is the position of the virtual viewpoint, and the direction of the person's face is the line of sight from the virtual viewpoint. Alternatively, the line-of-sight direction of the person may be set as the line-of-sight direction from the virtual viewpoint.

A viewpoint from behind the subject is a virtual viewpoint in which a position a predetermined distance behind the subject is the position of the virtual viewpoint, and a line of sight from the virtual viewpoint is a direction from that position toward the position of the subject. Alternatively, the line of sight direction from the virtual viewpoint may be determined according to the orientation of the subject. For example, when the subject is a person, the viewpoint from behind the subject is a virtual viewpoint that is a certain distance behind and a certain distance above the person's back, and the direction of the person's face is the line of sight from the virtual viewpoint. This is a virtual perspective.

A virtual viewpoint within a position on an arc centered on the subject is a position on a spherical surface defined by a predetermined radius centered on the subject's position as the virtual viewpoint position, and the direction is directed from that position to the subject's position. This is a virtual viewpoint where the direction is the line of sight direction from the virtual viewpoint.

For example, when the subject is a person, the position on a spherical surface defined by a predetermined radius centered on the person's position is the position of the virtual viewpoint, and the direction from that position to the position of the subject is the line of sight from the virtual viewpoint. This is a virtual perspective.

Step S33 thus functions as a virtual viewpoint image generation step that acquires a virtual viewpoint image of a viewpoint that has a predetermined relationship with the first image and uses it as a second image. Here, it is assumed that the virtual viewpoint image of a viewpoint that has a predetermined relationship with the first image is a virtual viewpoint image at the same time (imaging timing) as the viewpoint of the first image. Further, in this embodiment, the viewpoint having a predetermined relationship with the first image is a viewpoint having a predetermined angular relationship or a predetermined positional relationship with the viewpoint of the first image, as described above.

Then, in step S34, the CPU 111 associates the second image with the second surface 202 of the digital content 200. Note that the second image may be displayed for the operator's confirmation. At this time, the main image associated with the first surface 201 and the second image associated with the second surface 202 are synchronized so that they are images captured at the same time (time code) as described above. shall be controlled. In this way, in steps S31 to S34, the first image is associated with the first surface of the three-dimensional digital content, which will be described later, and the virtual viewpoint image of the virtual viewpoint having a predetermined relationship with the first image is associated with the second surface. 202. Further, steps S31 to S34 function as a content generation step (content generation means).

Next, in step S35, the CPU 111 determines whether an operation to change the viewpoint of the second image displayed on the second screen 202 has been performed via the operation unit 116. That is, the operator may change the viewpoint of the second image to be displayed on the second screen by, for example, selecting a camera image from a desired viewpoint from among the plurality of cameras 1 while watching a sports scene that changes from moment to moment. .

Alternatively, by specifying a desired viewpoint among the virtual viewpoints to the image generation unit 3, a virtual viewpoint image from that viewpoint may be obtained. In step S35, if such a viewpoint changing operation has been performed, the answer is Yes, and the process proceeds to step S36.

In step S36, the CPU 111 selects a viewpoint image after the viewpoint has been changed from among the plurality of cameras 1, or obtains a virtual viewpoint image after the viewpoint has been changed from the image generation unit 3. The virtual viewpoint image may be acquired by acquiring a virtual viewpoint image that has been generated in advance, or by newly generating a virtual viewpoint image based on a changed viewpoint. The acquired image is then set as a second image, and the process moves to step S34, where it is associated with the second surface. In this state, the first image, the second image, and the accompanying data are associated with the first to third pages of the digital content 200, respectively, on the display unit 115. The state in which the first image, second image, and accompanying data are associated with the first to third surfaces, respectively, may be confirmed by display. In this case, the surface numbers may also be displayed so that it can be seen which surfaces are the first to third surfaces.

If there is no viewpoint change in step S35, the process proceeds to step S37, and the CPU 111 determines whether or not to add an NFT to the digital content 200. For this purpose, for example, a display image (GUI) is displayed on the display unit 115 asking whether or not to add NFT to the digital content 200. If the operator selects to add an NFT, it is determined and the process proceeds to step S38, where the digital content 200 is provided with an NFT and encrypted, and then the process proceeds to step S39.

If the determination in step S37 is No, the process directly advances to step S39.

Note that the digital content 200 in step S37 may be a stereoscopic image having a shape as shown in FIGS. 4(B) and 4(C). Further, in the case of a polyhedron, it is not limited to a hexahedron as shown in FIG. 4(A), but may be an octahedron, for example.

In step S39, the CPU 111 determines whether to end the flow for generating the digital content 200 in FIG. 3. If the operator has not operated the operation unit 116 to end the process, the process returns to step S31 and repeats the above process, and if the process has ended, the flow in FIG. 3 ends. Note that even if the operator does not operate the operation unit 116 to terminate the process, the process may be automatically terminated after a predetermined period (for example, 30 minutes) has elapsed since the last operation of the operation unit 116.

Note that FIGS. 4(B) and 4(C) are diagrams showing modified examples of the digital content 200, and FIG. 4(B) shows the digital content 200 of FIG. 4(A) in the form of a sphere. A first image is displayed on the first surface 201, which is the spherical surface on the left side of the sphere 200 when viewed from the front, and a second image is displayed on the second surface 202, which is the spherical surface on the right side. Additionally, the above-mentioned accompanying data is displayed on the third surface 203, which is the upper spherical surface.

FIG. 4C is a diagram showing an example in which each plane of the digital content 200 in FIG. 4A is a curved surface with a desired curvature. In this way, the digital content in this embodiment may be displayed using a sphere as shown in FIGS. 4(B) and 4(C), a cube with a spherical surface, or the like.

(Embodiment 2)
Next, Embodiment 2 will be described using FIG. 5.

FIG. 5 is a flowchart for explaining the operation flow of the image processing system 100 according to the second embodiment. Note that each step in the flowchart of FIG. 5 is performed by the CPU 111 as a computer of the image processing system 100 executing a computer program stored in a memory such as the ROM 112 or the auxiliary storage device 114.

Note that in FIG. 5, steps with the same reference numerals as in FIG. 3 are the same processes, and a description thereof will be omitted.

In step S51 in FIG. 5, the CPU 111 obtains from the image generation unit 3 a camera image from a viewpoint specified by the operator or a virtual viewpoint image from a virtual viewpoint specified by the operator. The acquired image is then set as a second image. Other than that, the flow is the same as that of FIG. 3.

In the first embodiment, a second image having a predetermined relationship (different by a predetermined angle) from the main image (first image) is acquired. However, in the second embodiment, the operator selects a desired camera or obtains a virtual viewpoint image of a desired viewpoint of a desired subject as the second image.

Note that the camera image or virtual viewpoint image selected by the operator in step S51 includes, for example, an image from a bird's-eye view diagonally above the sports venue, or an image from a viewpoint diagonally from below. In this way, in the second embodiment, the operator can select the virtual viewpoint image displayed on the second surface.

Further, the virtual viewpoint image selected by the operator in step S51 may be a virtual viewpoint image taken from a position distant from the subject, such as a zoomed-out virtual viewpoint image.

In addition, camera images generated in the past, virtual viewpoint images generated based on them, etc. are stored in the storage unit 5, and they are read out and stored as the first image, second image, and accompanying data, respectively. It may be displayed on three sides.

Note that a step may be inserted between step S38 and step S39 in which the CPU 111 automatically switches to the default stereoscopic image display after a predetermined period (for example, 30 minutes) has elapsed since the last operation on the operation unit 116, for example. The default 3D image display is, for example, the main image on the first page, the accompanying data on the third page, and the camera image or virtual viewpoint image from the most frequent viewpoint based on past statistics on the second page. All you have to do is display it.

(Embodiment 3)
Embodiment 3 will be described using FIGS. 6 to 8. FIG. 6 is a flowchart for explaining the operation flow of the image processing system 100 of the third embodiment, FIG. 7 is a flowchart continuing from FIG. 6, and FIG. 8 is a flowchart continuing from FIGS. 6 and 7. Note that each step in the flowcharts of FIGS. 6 to 8 is performed by the CPU 111 as a computer of the image processing system 100 executing a computer program stored in a memory such as the ROM 112 or the auxiliary storage device 114.

In the third embodiment, when the operator selects the number of virtual viewpoints from 1 to 3, the display of the first to third surfaces of the digital content 200 is automatically switched accordingly.

In step S61, the operator selects the number of virtual viewpoints from 1 to 3, and the CPU 111 accepts it. In step S62, the CPU 111 acquires the selected number of virtual viewpoint images from the image generation unit 3.

At this time, a representative virtual viewpoint is automatically selected. That is, a scene is analyzed, and for that scene, for example, based on past statistics, the most frequently used virtual viewpoint is set as the first virtual viewpoint, the next most frequently used virtual viewpoint is set as the second virtual viewpoint, and the next virtual viewpoint is set as the second virtual viewpoint. A virtual viewpoint that is frequently used is set as a third virtual viewpoint. Note that the second virtual viewpoint may be set in advance to be different from the first virtual viewpoint by, for example, +90°, and the third virtual viewpoint may be set to be different from the first virtual viewpoint by, for example, −90°. Here, +90° and -90° are examples, and the angles are not limited to these.

In step S63, the CPU 111 determines whether the number of selected virtual viewpoints is 1 or not, and if it is 1, the process proceeds to step S64. In step S64, the CPU 111 acquires the main image from the main camera of the plurality of cameras 1 and associates it with the first page 201 of the digital content 200.

Then, in step S65, the CPU 111 associates the accompanying data with the third page 203 of the digital content 200. The accompanying data may be the same as the accompanying data displayed in step S32 of FIG. 3 of the first embodiment, such as the name of the player who shot at the goal.

In step S66, the CPU 111 associates the first virtual viewpoint image from the first virtual viewpoint with the second surface of the digital content 200, and then proceeds to step S81 in FIG. 8.

If the determination in step S63 is No, the CPU 111 determines in step S67 whether the number of selected virtual viewpoints is 2 or not, and if it is 2, the process proceeds to step S68.

In step S68, the CPU 111 associates the accompanying data with the third page 203 of the digital content 200. The accompanying data may be the same as the accompanying data associated in step S65, such as the name of the player who shot at the goal.

Then, in step S69, the CPU 111 associates the first virtual viewpoint image from the first virtual viewpoint with the first surface 201 of the digital content 200. Further, the second virtual viewpoint image from the aforementioned second virtual viewpoint is associated with the second surface 202 of the digital content 200. Thereafter, the process advances to step S81 in FIG.

If the determination in step S67 is No, the process advances to step S71 in FIG. In step S71, the CPU 111 determines whether or not the operator has selected to associate accompanying data with the third surface 203. If Yes, the process advances to step S72; if No, the process advances to step S73.

In step S72, the CPU 111 associates the virtual viewpoint image from the first virtual viewpoint with the first surface 201 of the digital content 200, the virtual viewpoint image from the second virtual viewpoint with the second surface 202, and the virtual viewpoint image from the second virtual viewpoint. A virtual viewpoint image from a third virtual viewpoint is associated with the three surfaces. Thereafter, the process advances to step S81 in FIG.

If No in step S71, the CPU 111 associates the accompanying data with the third page 203 of the digital content 200 in step S73. The accompanying data may be the same as the accompanying data associated in step S65, such as the name of the player who shot at the goal.

Then, in step S74, the CPU 111 associates the first virtual viewpoint image from the first virtual viewpoint with the first surface 201 of the digital content 200. Further, in step S75, the CPU 111 displays the second virtual viewpoint image from the second virtual viewpoint and the third virtual viewpoint image from the third virtual viewpoint side by side on the second surface 202 of the digital content 200. Correspond as possible. That is, the second surface 202 is divided into two areas displaying a second virtual viewpoint image and a third virtual viewpoint image, and the virtual viewpoint images are associated with each area. Thereafter, the process advances to step S81 in FIG.

In step S81 of FIG. 8, the CPU 111 determines whether to add an NFT to the digital content 200. For this purpose, for example, a display image (GUI) that asks whether to add NFT to the digital content 200 is displayed on the display unit 115. If the operator selects to add an NFT, the process proceeds to step S82, where the digital content 200 is provided with an NFT and encrypted, and then the process proceeds to step S83.

If the determination in step S81 is No, the process directly advances to step S83. Note that the digital content 200 in step S81 may have a shape as shown in FIGS. 4(B) and 4(C), as described above.

In step S83, the CPU 111 determines whether or not to end the flow of FIGS. 6 to 8, and if the operator has not operated the operation unit 116 to end the flow, the process proceeds to step S84.

In step S84, the CPU 111 determines whether the number of virtual viewpoints has been changed. If the number has been changed, the process returns to step S61. If the number has not been changed, the process returns to step S62. If the determination in step S83 is Yes, the flow of FIGS. 6 to 8 ends.

In the third embodiment, an example has been described in which when the operator selects the number of virtual viewpoints from 1 to 3, the images associated with the first to third surfaces of the digital content 200 are automatically switched accordingly. However, the operator may select the number of camera images to be associated with a surface forming the digital content 200 from among images taken by a plurality of cameras. Then, a predetermined camera may be automatically selected from among the plurality of cameras 1 in accordance with this, and the camera images on the first to third surfaces of the digital content 200 may be automatically associated with each other. Note that the number of viewpoints does not have to be three at most. For example, the number of viewpoints may be determined within a range that maximizes the number of surfaces constituting the digital content or the number of surfaces to which images can be associated. Furthermore, if multiple images can be associated with one surface, the maximum number of viewpoints can be further increased.

Further, between step S82 and step S83, a step is inserted in which the CPU 111 automatically switches to the content consisting of the default stereoscopic image display after a predetermined period (for example, 30 minutes) has elapsed since the last operation on the operation unit 116. Also good. In the default stereoscopic image display, for example, the main image is displayed on the first screen, and the camera image or virtual viewpoint image from the most frequently used viewpoint according to past statistics is displayed on the second screen. For example, the third page may contain accompanying data.

As described above, in the third embodiment, in steps S69, S72, and S95, it is possible to associate a virtual viewpoint image different from the virtual viewpoint image displayed on the second surface with the first surface.

(Embodiment 4)
Next, Embodiment 4 will be described using FIGS. 9 and 10. Note that in this embodiment, the system configuration is the same as the configuration described in Embodiment 1, so a description thereof will be omitted. Furthermore, the hardware configuration of the system is also the same as that in FIG. 2, and its description will be omitted.

This embodiment is a display image (GUI) that displays, on a user device, a three-dimensional digital content generated by any of the methods of embodiments 1 to 3. The user device may be, for example, a PC, a smartphone, or a tablet terminal with a touch panel (not shown). In this embodiment, a tablet terminal having a touch panel will be described as an example. This GUI is generated by the image processing system 100 and sent to the user device. Note that this GUI may be generated by the user device that has acquired the necessary information.

The image processing system includes a CPU, a ROM, a RAM, an auxiliary storage device, a display section, an operation section, a communication I/F, a bus, etc. (not shown). The CPU controls the entire image processing system using computer programs stored in the ROM, RAM, auxiliary storage, and the like.

The image processing system processes three-dimensional digital content, images obtained by capturing images from different directions using multiple imaging devices (cameras), virtual perspective images based on a specified virtual viewpoint, and images associated with the virtual perspective images. The system identifies information regarding the subject included in the captured sound information, captured image, and virtual viewpoint video.

The present embodiment will be described using, as an example, a three-dimensional digital content generated in the third embodiment, in which the number of virtual viewpoints is three, and three virtual viewpoint images are associated with the second surface. The three virtual viewpoint images are videos, and are hereinafter referred to as virtual viewpoint videos. In this embodiment, the three-dimensional digital content is a hexahedron, but it may also be a sphere or an octahedron.

The sound information associated with the virtual viewpoint video is sound information acquired at the venue at the time of image capture. Alternatively, sound information corrected based on the virtual viewpoint may be used. Sound information corrected based on the virtual viewpoint is, for example, sound information acquired at the venue during image capture that has been adjusted so that it can be heard when facing the line of sight from the virtual viewpoint at the position of the virtual viewpoint. It is sound information. Note that sound information may be prepared separately.

FIG. 9 is a diagram showing a graphic user interface (GUI) of this embodiment displayed on a user device. A first area 911 and a second area 912 exist in the GUI image. In the first area 911, a first display area 901, a second display area 902, a third display area 903, a fourth display area 904, and a fifth display display an image indicating information associated with three-dimensional digital content. An area 905 and a sixth display area 906 are included. Images and information assigned to each display area are displayed. Within the second area 912, there is a seventh display area 907 that displays an image showing information associated with the display area selected by the user from among the first display area 901 to the sixth display area 906. Note that the images indicating information associated with the first display area 901 to the sixth display area 906 may be still images or videos.

FIG. 9 shows an example in which the second display area 902 to which three virtual viewpoint videos are associated is selected. When display areas associated with three virtual viewpoint videos are selected, GUI 908, GUI 909, and GUI 910 corresponding to the virtual viewpoint videos of each viewpoint are displayed in the second area. Note that the GUI 908, GUI 909, and GUI 910 may be displayed in a superimposed manner on the seventh display area 907.

In this embodiment, each surface of the three-dimensional digital content is associated with each display area. That is, an image indicating information associated with each side of the digital content is displayed in each display area. Note that an image indicating information associated with the digital content may be displayed in each display area regardless of the shape of the digital content.

Note that the number of display surfaces and display areas for digital content may be different. For example, for hexahedral digital content, only the first to fourth display areas 901 to 904 may be displayed on the user device. Part of the information associated with the digital content is displayed in the first display area 901 to the third display area 903, and the information associated with the display area selected by the user is displayed in the fourth display area 904.

Information identified from three-dimensional digital content is associated with each display area, and an image representing the identified information is displayed. In this embodiment, the subject is a basketball player, and a main image showing the player is displayed in the first display area 901. The image displayed in the first display area 901 may be a virtual viewpoint image or a photographed image. In the second display area 902, an image showing three virtual viewpoint videos related to the player displayed in the main image and an icon 913 showing the virtual viewpoint video are displayed in a superimposed manner. In the second display area 902, a virtual viewpoint image corresponding to a different viewpoint from the image displayed in the first display area 901 is displayed. An image showing information about the team to which the player displayed in the main image belongs is displayed in the third display area 903. In the fourth display area 904, an image showing performance information of the player displayed in the main image in the season in which the main image was captured is displayed. An image showing the final score of the match at the time of shooting is displayed in the fifth display area 905. An image showing copyright information of the digital content is displayed in the sixth display area 906.

If the information associated with the first display area 901 to the sixth display area 906 includes information indicating a video, an illustration or an icon may be displayed superimposed on the image in the display area corresponding to the video. In this case, different icons are used for a video generated from an image captured by one imaging device and a virtual viewpoint video generated from virtual viewpoint images generated by a plurality of imaging devices. In this embodiment, an icon 913 is displayed superimposed on the image showing the virtual viewpoint video in the second display area 902. Further, when the second display area 902 is selected by the user, an icon 913 is displayed on the virtual viewpoint video displayed in the seventh display area 907. Note that illustrations and icons may be placed around the display area.

A plurality of images or a plurality of videos may be associated with one display area among the first display area 901 to the sixth display area 906. In this embodiment, a case will be described in which three virtual viewpoint videos having different viewpoints are associated with the second display area 902. In this case, virtual viewpoint videos having different viewpoints are associated with the GUI 908, GUI 909, and GUI 910, respectively. A virtual viewpoint image of the subject viewpoint (viewpoint 1) is associated with the GUI 908. A virtual viewpoint video corresponding to a viewpoint from behind the subject (viewpoint 2) is associated with the GUI 909. The GUI 910 is associated with a virtual viewpoint image of a virtual viewpoint (viewpoint 3) among positions on a spherical surface centered on the subject. Note that if multiple images or multiple videos are not associated with one display area, GUI 908, GUI 909, and GUI 910 do not need to be displayed.

An initial image is set in the seventh display area 907 as information to be displayed before the user selects the sixth display area 906 from the first display area 901. The initial image may be an image associated with the first display area 901 to the sixth display area 906, or may be a different image from the image associated with the first display area 901 to the sixth display area 906. . In this embodiment, the main image in the first display area 901 is set as the initial image.

FIG. 10 is a flowchart for explaining the operation flow of the image processing system in this embodiment. Specifically, this is a process executed by the CPU 111 in FIG. In step S1001, content information associated with the first to sixth sides of the three-dimensional digital content is identified.

In step S1002, the content information specified in step S1001 is associated with the first display area 901 to the sixth display area 906. Thereafter, images indicating the associated information are displayed from the first display area 901 to the sixth display area 906. Further, a preset initial image is displayed in the seventh display area 907. In this embodiment, the main image in the first display area 901 is displayed in the seventh display area 907 as an initial image.

In step S1003, it is determined whether a predetermined time (for example, 30 minutes) has elapsed since the latest input was received. If Yes, the process advances to step S1017. If No, the process advances to step S1004.

In step S1004, it is determined whether an input by the user to select any one of the first display area 901 to the sixth display area 906 has been received. If Yes, the steps to proceed will differ depending on the received input. If an input to select the first display area 901 is accepted, the process advances to step S1005. If an input to select the second display area 902 is accepted, the process advances to step S1006. If an input to select the third display area 903 is accepted, the process advances to step S1007. If an input to select the fourth display area 904 is accepted, the process advances to step S1008. If an input to select the fifth display area 905 is accepted, the process advances to step S1009. If an input to select the sixth display area 906 is accepted, the process advances to step S1010. If No, the process returns to step S1003.

In step 1005, the main image showing the player corresponding to the first display area 901 is displayed in the seventh display area 907. Note that if the main image showing the player associated with the first display area 901 is already displayed in the seventh display area 907, the process returns to step S1003 while the main image is displayed in the seventh display area 907. If information related to the display area selected by the user is already displayed in the seventh display area 907, the same applies to steps 1007 to 1010, so the description thereof will be omitted. Further, if the main image is a video and the main image is already displayed in the seventh display area 907, the video may be played again from the default playback time, or the displayed video may be played as is. Good too.

In step 1006, a virtual viewpoint video related to the player corresponding to the second display area 902 is displayed in the seventh display area 907. Note that when a plurality of virtual viewpoint videos are associated with the second display area 902, a preset virtual viewpoint video is displayed in the seventh display area 907. In this embodiment, a virtual viewpoint image from the subject viewpoint (viewpoint 1) is displayed in the seventh display area 907. After displaying the preset virtual viewpoint video in the seventh display area 907, the process advances to step S1011.

In step S1011, it is determined whether a predetermined time (for example, 30 minutes) has elapsed since the latest input was received. If Yes, the process advances to step S1017. If No, the process advances to step S1012.

In step S1012, it is determined whether an input for selecting a virtual viewpoint video of a virtual viewpoint that the user wants to see from among a plurality of virtual viewpoint videos is received. Specifically, a GUI indicating each virtual viewpoint is displayed as in the seventh display area 907 in FIG. 9, and when the user selects the GUI, the virtual viewpoint video of each viewpoint is selected. If Yes, proceed to the next step depending on the selected GUI. If the GUI 908 is selected, the process proceeds to step S1013, if the GUI 909 is selected, the process proceeds to step S1014, and if the GUI 910 is selected, the process proceeds to step S1015. If No, the process advances to step S1016.

Note that the virtual viewpoint video of each viewpoint may be selected by a flick operation or a touch operation on the seventh display area 907 without providing a GUI indicating each virtual viewpoint. Further, when a plurality of virtual viewpoint videos are associated with the second display area 902, the plurality of virtual viewpoint videos may be connected to form one virtual viewpoint video, and the virtual viewpoint video may be continuously played. In that case, the process proceeds to step S1016 without going through step S1012 for selecting a virtual viewpoint.

In step S1013, a virtual viewpoint image of the subject viewpoint (viewpoint 1) is displayed in the seventh display area 907. After that, the process returns to step S1011. If the virtual viewpoint video from the subject's viewpoint is already displayed in the seventh display area 907, the video may be played back from the predetermined playback time, or the displayed video may be played back as is. If it is already displayed in the seventh display area 907, the same applies to steps S1014 and S1015, so the description thereof will be omitted below.

In step S1014, a virtual viewpoint video corresponding to a viewpoint from behind the subject (viewpoint 2) is displayed in the seventh display area 907. After that, the process returns to step S1011.

In step S1015, a virtual viewpoint image of a virtual viewpoint (viewpoint 3) among positions on a spherical surface centered on the subject is displayed in the seventh display area 907. After that, the process returns to step S1011.

In step S1016, it is determined whether an input by the user to select any one of the first display area 901 to the sixth display area 906 has been received. If Yes, the same process as step S1004 is performed. If No, the process returns to step S1011.

In step S1007, team information to which the player corresponding to the third display area 903 belongs is displayed in the seventh display area 907.

In step S1008, this season's performance information of the player corresponding to the fourth display area 904 is displayed in the seventh display area 907.

In step S1009, the final score of the match corresponding to the fifth display area 905 is displayed in the seventh display area 907.

In step S1010, copyright information corresponding to the sixth display area 906 is displayed in the seventh display area 907.

In step S1017, the initial image is displayed in the seventh display area. In this embodiment, the main image in the first display area 901 is displayed in the seventh display area 907 as an initial image. After that, the processing flow ends.

(Embodiment 5)
Next, Embodiment 5 will be described using FIGS. 11 and 12. Note that in this embodiment, the system configuration is the same as the configuration described in Embodiment 1, so a description thereof will be omitted. Furthermore, the hardware configuration of the system is also the same as that in FIG. 2, and its description will be omitted.

FIG. 11 is a diagram showing the graphic user interface (GUI) of this embodiment. As in the fourth embodiment, a three-dimensional digital content that is generated in the third embodiment, has three virtual viewpoints, and has three virtual viewpoint images associated with the second surface will be described as an example. Further, unlike the fourth embodiment, the number of display areas displayed in the first area 1107 is different from the number of pages of digital content. Specifically, while the number of display areas displayed in the first area 1107 is five, the number of pages of digital content is six. This GUI is generated by the image processing system 100 and sent to the user device. Note that this GUI may be generated by the user device that has acquired the necessary information.

In this embodiment, the digital content includes three virtual viewpoint videos with different viewpoints, and the virtual viewpoint videos of each viewpoint are associated with the second display area 1102 to the fourth display area 1104, respectively. As in the fourth embodiment, the three virtual viewpoints are a virtual viewpoint image of the subject's viewpoint (viewpoint 1), a viewpoint from behind the subject (viewpoint 2), and a virtual viewpoint within the position on the spherical surface centered on the subject ( Viewpoint 3). In order to associate the three virtual viewpoint videos from the second display area 1102 to the fourth display area 1104, an icon 1109 indicating the virtual viewpoint video is also superimposedly displayed from the second display area 1102 to the fourth display area 1104.

The first display area 1101 is associated with an image showing a player, and the fifth display area 1105 is associated with copyright information. The first display area 1101 may contain a virtual viewpoint video or a photographed image from a different viewpoint from the virtual viewpoint images from the second display area 1102 to the fourth display area 1104. Note that the information displayed in the display area is not limited to these, and may be any information that is associated with digital content.

FIG. 12 is a flowchart for explaining the operation flow of the image processing system 100 of this embodiment. Note that each step in the flowchart of FIG. 12 is performed by the CPU 111 as a computer of the image processing system 100 executing a computer program stored in a memory such as the ROM 112 or the auxiliary storage device 114. Note that in FIG. 12, steps with the same reference numerals as in FIG. 10 are the same processes, and a description thereof will be omitted.

If an input to select the second display area 1102 is received in step S1004 of FIG. 12, the process advances to step S1201. If an input to select the third display area 1103 is accepted, the process advances to step S1202. If an input to select the fourth display area 1104 is accepted, the process advances to step S1203.

In step S1201, a virtual viewpoint image of the subject viewpoint (viewpoint 1) is displayed in the sixth display area 1106. After that, the process returns to step S1003.

In step S1202, a virtual viewpoint video corresponding to a viewpoint from behind the subject (viewpoint 2) is displayed in the sixth display area 1106. After that, the process returns to step S1003.

In step S1203, a virtual viewpoint image of a virtual viewpoint (viewpoint 3) among positions on a spherical surface centered on the subject is displayed in the sixth display area 1106. After that, the process returns to step S1003.

(Embodiment 6)
Next, Embodiment 6 will be described using FIGS. 13 and 14. Note that in this embodiment, the system configuration is the same as the configuration described in Embodiment 1, so a description thereof will be omitted. Furthermore, the hardware configuration of the system is also the same as that in FIG. 2, and its description will be omitted.

FIG. 13 is a diagram showing a graphic user interface (GUI) according to the sixth embodiment. The present embodiment will be described using as an example the digital content in a three-dimensional shape that is generated in the third embodiment, has six virtual viewpoints, and has six virtual viewpoint images associated with the second surface. The six virtual viewpoint images are videos, and are hereinafter referred to as virtual viewpoint videos. This GUI is generated by the image processing system 100 and sent to the user device. Note that this GUI may be generated by the user device that has acquired the necessary information.

This embodiment is different from the fourth embodiment in that the number of display areas displayed in the first area 1307 is different from the number of pages of digital content. Specifically, while the number of display areas displayed in the first area 1307 is three, the number of pages of digital content is six. Additionally, six virtual viewpoint videos are associated with the second display area 1302.

The first display area 1301 is associated with a main image showing a player, and the third display area 1303 is associated with copyright information. The first display area 1301 may contain a virtual viewpoint video or a photographed image from a different viewpoint from the virtual viewpoint image in the second display area 1302. Note that the information displayed in the display area is not limited to these, and may be any information that is associated with digital content.

This embodiment differs from Embodiments 4 and 5 in that the second area 1308 includes a fourth display area 1304, a fifth display area 1305, and a sixth display area 1306. When the user selects any one of the first display area 1301 to the third display area 1303, an image corresponding to the selected display area is displayed in the fifth display area.

The fifth display area 1305 is always displayed in the second area 1308. On the other hand, the fourth display area 1304 and the sixth display area 1306 are displayed in the second area 1308 when the virtual viewpoint video is displayed in the fifth display area.

The display area located at the center of the second area 1308 and the other display areas have different shapes. Specifically, the fourth display area 1304 and the sixth display area 1306 are different in shape or size from the fifth display area 1305. In this embodiment, the fifth display area has a rectangular shape, whereas the fourth display area and the sixth display area have a trapezoidal shape. By doing so, the visibility of the fifth display area 1305 located at the center of the second area 1308 can be improved.

Each of the six virtual viewpoint images has a different virtual viewpoint. There are three subjects, three virtual viewpoint images with the positions of the three subjects as the virtual viewpoint positions, and three virtual viewpoint images with the virtual viewpoint positions a certain distance behind and a certain distance above the positions of the three subjects. This is a perspective video. For example, in the case of a basketball game, where player A plays offense, player B plays defense, and the basketball is the subject, the following example may be given. One is a first virtual viewpoint video in which the position of Player A's face is the position of the virtual viewpoint, and the direction of Player A's face is the line of sight direction from the virtual viewpoint. Another method is to set the virtual viewpoint at a position a certain distance behind (e.g., 3 m behind) and a certain distance (e.g., 1 m higher) from the position of player A's face, and to include player A within the angle of view. This is a second virtual viewpoint video in which the direction set in is the line-of-sight direction from the virtual viewpoint. Another one is a third virtual viewpoint video in which the position of player B's face is the position of the virtual viewpoint, and the direction of player B's face is the line of sight direction from the virtual viewpoint. Another method is to set the virtual viewpoint position a certain distance behind and a certain distance higher than the position of player B's face, and set the direction set to include player B within the angle of view as the viewing direction from the virtual viewpoint. This is the fourth virtual viewpoint video. Another one is a fifth virtual viewpoint video in which the position of the center of gravity of the basketball is the position of the virtual viewpoint, and the direction of progress of the basketball is the line of sight from the virtual viewpoint. Another one is a sixth virtual viewpoint image in which the virtual viewpoint is a certain distance behind and a certain distance above the center of gravity of the basketball, and the virtual viewpoint is in the direction of movement of the basketball. Note that the position a certain distance behind and a certain distance higher than the position of the subject may be determined based on the shooting scene, or may be determined using the ratio of the subject to the angle of view of the virtual viewpoint image. The viewing direction from the virtual viewpoint is set based on any one of the posture of the subject, the direction of movement of the subject, and the position of the subject in the angle of view.

In this embodiment, it is assumed that the six virtual viewpoint videos have the same playback time. Note that the virtual viewpoint videos may have different playback times.

When the second display area 1302 corresponding to six virtual viewpoint videos is selected, the fourth display area 1304 and the sixth display area 1306 are displayed in the second area 1308. The three display areas displayed in the second area 1308 are associated with virtual viewpoint videos showing three objects, respectively. Specifically, the fourth display area 1304 displays a first virtual viewpoint video and a second virtual viewpoint video with player A as the subject, and the fifth display area 1305 displays a third virtual viewpoint video and a fourth virtual viewpoint with player B as the subject. A fifth virtual viewpoint video and a sixth virtual viewpoint video with a basketball as the subject are associated with the viewpoint video and the sixth display area 1306.

The images displayed in each display area are not all played back at the same time. Only the video displayed in the display area located at the center of the second area 1308 is played back. In this embodiment, only the virtual viewpoint video displayed in the fifth display area 1305 is played back.

FIG. 14 is a flowchart for explaining the operation flow of the image processing system 100 according to the sixth embodiment. Note that each step in the flowchart of FIG. 5 is performed by the CPU 111 as a computer of the image processing system 100 executing a computer program stored in a memory such as the ROM 112 or the auxiliary storage device 114. Note that in FIG. 14, steps with the same reference numerals as in FIG. 10 are the same processes, and a description thereof will be omitted.

In step S1401, preset virtual viewpoint images are displayed in the fourth display area 1304, the fifth display area 1305, and the sixth display area 1306. In this embodiment, three virtual viewpoint videos are displayed with the positions of three subjects as the virtual viewpoint positions. Specifically, the fourth display area 1304 displays a first virtual viewpoint image with the position of player A's face as the virtual viewpoint position, and the fifth display area 1305 displays a first virtual viewpoint image with the position of player B's face as the virtual viewpoint position. In the third virtual viewpoint video and the sixth display area 1306, a fifth virtual viewpoint video is displayed in which the center of gravity of the basketball is the virtual viewpoint position. At this time, only the fifth display area 1305 located at the center of the second area 1308 plays back the video, and the fourth display area 1304 and the sixth display area 1306 do not play back the video. After displaying the preset virtual viewpoint video in the fourth display area 1304, fifth display area 1305, and sixth display area 1306, the process advances to step S1011.

In step S1402, it is determined whether an operation to change the subject of the virtual viewpoint video displayed in the fifth display area 1305 has been input. Specifically, it is determined whether an operation to switch to a virtual viewpoint image of a different subject by a horizontal slide operation on the fifth display area 1305 has been input. If Yes, proceed to the next step based on the slid direction. If input information for a leftward sliding operation is received, the process advances to step S1410. If input information for a rightward sliding operation is accepted, the process advances to step S1411. If No, the process advances to step S1405.

In step S1403, the virtual viewpoint video associated with each display area is associated again with the display area adjacent to the left. For example, when a slide operation to the left is received for the third virtual viewpoint video corresponding to the fifth display area 1305, the third virtual viewpoint video and the fourth virtual viewpoint video corresponding to the fifth display area 1305 are moved to the fifth virtual viewpoint video. It is associated with the fourth display area 1304 on the left side of the display area 1305. The fifth virtual viewpoint video and the sixth virtual viewpoint video associated with the sixth display area 1306 are associated with the fifth display area 1305 on the left side of the sixth display area 1306. In the second area 1308, since there is no display area to the left of the fourth display area 1304, there is a display area to the right of the first virtual viewpoint video and the second virtual viewpoint video corresponding to the fourth display area 1304. It is associated with the sixth display area 1306 that is not displayed. After the association is performed again, the position of the virtual viewpoint with respect to the subject among the virtual perspective images associated with the fifth display area 1305 will be the same as the virtual perspective image that was being played in the fifth display area 1305 before the association. Play the same virtual viewpoint video. For example, if the virtual viewpoint video that was being played in the fifth display area 1305 before the mapping was the third virtual viewpoint video, after the mapping, the fifth virtual viewpoint video and the sixth virtual viewpoint video are displayed in the fifth display area 1305. Images are associated. Since the third virtual viewpoint video is a virtual viewpoint video with the subject's position as the virtual viewpoint, the fifth virtual viewpoint video, which is also a virtual viewpoint video with the subject's position as the virtual viewpoint, is displayed in the fifth display area. Is displayed. By doing so, the user can intuitively switch between virtual viewpoint videos of different subjects. After performing the above processing, the process returns to step S1011.

In step S1404, the virtual viewpoint video associated with each display area is associated again with the display area on the right. For example, when a slide operation to the right is received for the third virtual viewpoint video corresponding to the fifth display area 1305, the third virtual viewpoint video and the fourth virtual viewpoint video corresponding to the fifth display area 1305 are It is associated with the sixth display area 1306 on the right side of the display area 1305. The first virtual viewpoint video and the second virtual viewpoint video associated with the fourth display area 1304 are associated with the fifth display area 1305 on the right side of the fourth display area 1304. In the second area 1308, since there is no display area on the right side of the sixth display area 1306, there is a display area on the left side of the fifth virtual viewpoint video and the sixth virtual viewpoint video corresponding to the sixth display area 1306. It is associated with the fourth display area 1304 that is not displayed. After the association is performed again, the position of the virtual viewpoint with respect to the subject among the virtual perspective images associated with the fifth display area 1305 will be the same as the virtual perspective image that was being played in the fifth display area 1305 before the association. Play the same virtual viewpoint video. After performing the above processing, the process returns to step S1408. After performing the above processing, the process returns to step S1011.

In step S1405, it is determined whether an operation to change the virtual viewpoint position of the virtual viewpoint video displayed in the fifth display area 1305 has been input. Specifically, an input to switch to a virtual viewpoint video of the same subject but with a different virtual viewpoint position is accepted by a double tap operation on the fifth display area 1305. If Yes, the process advances to step S1406; if No, the process advances to step S1016.

In step S1406, processing is performed to change the position of the subject in the virtual viewpoint video. Specifically, the video is switched to a virtual viewpoint video of the same subject but with a different virtual viewpoint position. For example, the fifth display area 1305 displays a third virtual viewpoint image with the virtual viewpoint position of player B's face and a third virtual viewpoint video with the virtual viewpoint position a certain distance behind and a certain distance higher than the position of player B's face. It is assumed that four virtual viewpoint images are associated with each other. If a double-tap operation is received while the third virtual viewpoint video is being displayed in the fifth display area 1305, a process of switching to the fourth virtual viewpoint video and displaying it in the fifth display area 1305 is performed. By doing this, it is possible to intuitively switch the position of the virtual viewpoint for the same subject. After performing the above processing, the process returns to step S1011.

Note that when switching the virtual viewpoint video using a slide operation or a double tap operation, the time code of the virtual viewpoint video being played in the fifth display area 1305 is recorded, and the time code of the virtual viewpoint video after switching is recorded. You can start playing from the time.

Note that in this embodiment, the double-tap operation is an operation for switching the position of the virtual viewpoint on the same subject, but other operations may be used. For example, it may be a pinch-in/pinch-out operation or a slide operation in the vertical direction on the fifth display area 1305.

In this embodiment, a plurality of virtual viewpoint videos showing the same subject are associated with one display area, but a plurality of virtual viewpoint videos showing the same subject may be associated with a plurality of display areas. Specifically, a seventh display area, an eighth display area, and a ninth display area may be newly provided above the fourth display area 1304, the fifth display area 1305, and the sixth display area 1306, respectively (not shown). ). In this case, the fourth display area 1304 has a first virtual viewpoint video, the fifth display area 1305 has a third virtual viewpoint video, the sixth display area 1306 has a fifth virtual viewpoint video, the seventh display area has a second virtual viewpoint video, A fourth virtual viewpoint video is displayed in the eighth display area, and a sixth virtual viewpoint video is displayed in the ninth display area. In the sixth embodiment, the tap gesture was used to switch between virtual viewpoint images showing the same subject, but in this case, the display area is switched by sliding in the vertical direction. By doing so, the user can intuitively operate the virtual viewpoint.

(Embodiment 7)
In the first embodiment, a second image having a predetermined relationship with the main image (first image) associated with the first page 201 of the digital content 200 is associated with the second page 202 of the digital content 200. . In this embodiment, a plurality of virtual viewpoint videos having the same time code are associated with each side of three-dimensional digital content. Specifically, an example will be described in which a virtual viewpoint image is associated with each surface of a three-dimensional digital content based on the line of sight direction of the virtual viewpoint with respect to the subject.

FIG. 15 is a diagram showing the image processing system 101 of this embodiment. Note that blocks similar to those in FIG. 1 are given the same numbers, and explanations thereof will be omitted.

The image generation unit 117 generates a correspondence relationship between the position of the virtual viewpoint, the line of sight direction from the virtual viewpoint, and the coordinates of the object based on the virtual viewpoint designated by the operation unit 116 and the coordinate information of the object displayed in the virtual viewpoint video. Analyze. The image generation unit 117 identifies the object of interest from the virtual viewpoint video seen from the virtual viewpoint specified by the operation unit 116. In this embodiment, the object that appears at the center of the virtual viewpoint video or is closest to the center is specified, but the present invention is not limited thereto. For example, the object that occupies the highest proportion of the virtual viewpoint video may be specified, or the object may be selected without generating the virtual viewpoint video. Next, the image generation unit 117 determines a photographing direction in which to photograph the identified object of interest, and generates a plurality of virtual viewpoints corresponding to each photographing direction. The shooting directions are top, bottom, left, right, front and back (front and back). Note that the plurality of generated virtual viewpoints correspond to the same time code as the virtual viewpoint specified by the operator. After generating a virtual viewpoint video corresponding to the generated virtual viewpoint, it is determined from which shooting direction each virtual viewpoint video captures the object of interest, top, bottom, left, right, front, back, front, back, etc. Add direction information. The photographing direction information is information indicating from which direction the object of interest was photographed. The method for determining the shooting direction is determined based on the positional relationship between the object centrally captured in the virtual viewpoint video and a predetermined point at the start of the virtual viewpoint video. Details will be explained with reference to FIG. 17.

The content generation unit 118 determines which side of the three-dimensional content the virtual viewpoint video received from the image generation unit 117 is to be associated with, based on the attached shooting direction information, and generates three-dimensional digital content.

FIG. 16 is a diagram illustrating each side of the three-dimensional digital content in this embodiment. In the three-dimensional digital content 1600, a surface 1601 is defined as the front surface, a surface 1602 as the right surface, a surface 1603 as the top surface, a surface 1604 as the left surface, a surface 1605 as the back surface, and a surface 1606 as the bottom surface.

FIG. 17 is a diagram illustrating shooting directions for athletes. On the court 1700, there are a goal 1701, a goal 1702, and a player 1703. It is assumed that player 1703 attacks toward goal 1701. Here, the direction connecting the player 1703 and the goal 1701, and the direction from the goal 1701 toward the player 1703 horizontally to the ground, is defined as the "front". Here, the method for determining the front will be explained. First, a line segment connecting the player 1703 and the goal 1701 is derived, with the goal being a predetermined point and the player being the center of gravity of the 3D model. Next, a surface that is perpendicular to the derived line segment and in contact with the 3D model of the player is derived, and this surface is defined as the front. After determining the front, the bounding box surrounding the player is determined. The top, bottom, right, left, and back surfaces are determined based on the front of the bounding box. In FIG. 17, the direction in which the player is viewed from the front is expressed by an arrow 1704, and the shooting direction information of the virtual viewpoint video created with this direction as the viewing direction from the virtual viewpoint is "front". The virtual viewpoint video to which the shooting direction information of "front" is attached is associated with the plane 1601 in FIG. 16. Similarly, the shooting direction information of the virtual viewpoint video created with the direction of the arrow 1706 taken from the right of the player as the viewing direction from the virtual viewpoint is "right side". The virtual viewpoint video to which the shooting direction information of “right side” is attached is associated with the 1602 screen in FIG. 16 . The shooting direction information of the virtual viewpoint video created with the direction of the arrow 1707 taken from the left of the player as the viewing direction from the virtual viewpoint is "left side." The virtual viewpoint video to which the shooting direction information of “left side” is attached is associated with the 1604th screen in FIG. 16 . The shooting direction information of the virtual viewpoint video created with the direction of the arrow 1708, which captures the player from above, as the viewing direction from the virtual viewpoint is "top". The virtual viewpoint video to which the shooting direction information of “top view” is attached is associated with the 1603 screen in FIG. 16 . The shooting direction information of the virtual viewpoint video created with the direction of the arrow 1709, which captures the player from below, as the viewing direction from the virtual viewpoint is "bottom surface". The virtual viewpoint video to which shooting direction information of "bottom surface" is attached is associated with screen 1606 in FIG. 16. The shooting direction information of the virtual viewpoint video created with the direction of the arrow 1705 that captures the player from behind as the viewing direction from the virtual viewpoint is "back". The virtual viewpoint video to which "rear" shooting direction information is attached is associated with screen 1605 in FIG. 16. In this embodiment, the direction is determined based on the relationship between the player and the goal at a specific moment, but the direction may be changed based on the relationship between the player and the goal as the player moves on the field.

Note that in this embodiment, the photographing direction is determined based on the positions of the players and the goal, but the present invention is not limited to this. For example, the direction in which the player is viewed from the player's advancing direction is set as "front", and the directions obtained by rotating the "front" direction by ±90 degrees on the XY plane are set as "left side" and "right side". Directions obtained by rotating the "front" direction by ±90 degrees on the YZ plane are set as "top" and "bottom". The direction obtained by rotating the "front" direction by +180 degrees or -180 degrees on the XY plane is set as the "back". In addition, as another example of setting the front direction, you can set the direction seen from the player's face as "front", or the direction seen from the goal position closest to the straight line in the player's direction of progress. It may also be set to "front".

Note that each time the player moves, the shooting direction may be changed depending on the positional relationship with the goal. For example, the front may be determined again when the front is moved a certain distance (for example, 3 meters) or more from a predetermined position, or the photographing direction may be changed after a certain period of time has elapsed. Alternatively, when comparing the initially determined front and the front calculated after movement, if the angle when the surface is viewed from above changes by 45 degrees or more, the front may be redefined. Furthermore, the front direction may be redefined each time based on the relationship between the player who received the pass and the goal position, using the timing when the ball is passed to another player as a trigger.

FIG. 18 is a diagram illustrating an example of three-dimensional digital content generated by the content generation unit 4. A plurality of virtual viewpoint images viewed from a plurality of virtual viewpoints corresponding to the same time code as the virtual viewpoint set by the operator correspond to each side of the three-dimensional digital content. By displaying the image in this manner, the user can intuitively understand from which position the object is photographed in the virtual viewpoint image.

FIG. 19 is a flowchart for explaining the operation flow of the image processing system 101 according to the seventh embodiment. Note that the same numbers are given to the same flows as S37 to S39 in FIG. 3, and the description thereof will be omitted.

In S1901, the image generation unit 117 acquires virtual viewpoint information indicating the position of a virtual viewpoint specified by the user via the operation unit 116 and the direction of line of sight from the virtual viewpoint.

In S1902, the image generation unit 117 identifies the object of interest in the virtual viewpoint image seen from the virtual viewpoint corresponding to the acquired virtual viewpoint information. In this embodiment, an object that appears at the center of the virtual viewpoint video or is closest to the center is identified.

In S1903, the image generation unit 117 determines the shooting direction for the object of interest. In this embodiment, the front surface is defined as a surface that is perpendicular to the straight line between the position of the object of interest and a predetermined position and that is in contact with the 3D model of the object of interest. Based on this front, the shooting directions for the top, bottom, left, right, and back are determined.

In S1904, the image generation unit 117 generates a plurality of virtual viewpoints corresponding to the plurality of shooting directions determined in S1903. In this embodiment, photographing directions corresponding to the front, back, top, bottom, left, and right of the object of interest are determined, so virtual viewpoints corresponding to each are generated. Note that the generated virtual viewpoint only needs to have the viewing direction from the virtual viewpoint set in the same direction as the photographing direction, and the object of interest does not need to be captured on the optical axis of the virtual viewpoint. Further, the position of the generated virtual viewpoint is set to a position separated by a predetermined distance from the position of the object of interest. In this embodiment, it is assumed that the virtual viewpoint is set at a position 3 meters away from the object of interest.

In S1905, the image generation unit 117 generates a virtual viewpoint image corresponding to the generated virtual viewpoint. Thereafter, photographing direction information indicating a photographing direction corresponding to the virtual viewpoint is added to the generated virtual viewpoint image.

In S1906, the image generation unit 117 determines whether virtual viewpoint videos have been generated for all the virtual viewpoints generated in S1904. If all virtual viewpoint videos have been generated, the generated virtual viewpoint videos are sent to the content generation unit 118, and the process advances to S1907. If all virtual viewpoint videos have not been generated, the process advances to S1905 and loops until all virtual viewpoint videos are generated.

In S1907, the content generation unit 118 associates the virtual viewpoint video with each surface of the three-dimensional digital content based on the received shooting direction information of the virtual viewpoint video.

In S1908, the content generation unit 118 determines whether the received virtual viewpoint video is associated with all sides of the three-dimensional digital content. If they are associated, the process advances to S37; if not, the process advances to S1907. Note that in this embodiment, it is assumed that the information is associated with all surfaces, but the present invention is not limited to this, and it may be made to be associated with a specific surface. In that case, in S1908, it is determined whether a virtual viewpoint image is associated with a specific surface.

Through the above processing, each surface of the three-dimensional digital content can be associated with a virtual viewpoint video according to the shooting direction. As a result, when a user viewing a virtual viewpoint video using digital content wants to switch virtual viewpoint videos, he or she can intuitively grasp the virtual viewpoint video corresponding to each side.

(Embodiment 8)
In the seventh embodiment, a plurality of virtual viewpoints corresponding to the same time code are generated based on a virtual viewpoint specified by the operator, and a virtual viewpoint video corresponding to the shooting direction of each virtual viewpoint is associated with each side of the digital content. I attached it. However, there may be cases where it is desired to associate a virtual viewpoint image seen from a virtual viewpoint specified by the operator with a plane corresponding to the shooting direction. In the eighth embodiment, the virtual viewpoint video viewed from the virtual viewpoint specified by the operator specifies from which shooting direction the object of interest is shot, and is associated with each side of the digital content according to the shooting direction.

FIG. 20 is a flowchart for explaining the operation flow of the image processing system 101 according to the eighth embodiment. Flows that are the same as S37 to S39 in FIG. 3, S1901 and S1902, S1907 and S1908 in FIG.

In step S2001, the image generation unit 117 generates a virtual viewpoint video based on the virtual viewpoint information acquired in S1901.

In step S2002, the image generation unit 117 determines the shooting direction for the object of interest for each frame of the virtual viewpoint video. For example, if the virtual viewpoint video has a total of 1000 frames, the number of frames to which "front" shooting direction information is added is 800 frames. It is assumed that the number of frames to which "rear" photographing direction information is added is 100 frames. It is assumed that the number of frames to which "left side" photographing direction information is added is 50 frames. It is assumed that the number of frames to which the shooting direction information of "right side" is added is 30 frames. It is assumed that there are 20 frames to which "top" photographing direction information is added. It is assumed that the number of frames to which the shooting direction information of "bottom surface" is added is 10 frames. In this way, shooting direction information is assigned to each frame with a different time code.

Note that in this embodiment, when a user views digital content, it is assumed that when frames with different shooting directions are displayed, the frame is rotated to the corresponding digital content plane. By doing so, it is possible to take advantage of the three-dimensional shape and give a sense of dynamism to the virtual viewpoint video.

Further, in the present embodiment, the surface to be associated with the three-dimensional digital content is determined in advance with respect to the photographing direction, but the present invention is not limited to this. For example, the surfaces to be associated with the three-dimensional digital content may be determined in order of the highest frame ratio of each virtual viewpoint video capturing the object. Specifically, the second to sixth directions are set in descending order of frame ratio. The virtual viewpoint video has 1000 frames, 800 frames from the front 1604, 100 frames from the back 1605, 50 frames from the left 1606, 30 frames from the right 1607, 20 frames from the top 1608, and 10 frames from the bottom 1609. shall be. In this case, the first direction is determined as the front, and the second to sixth directions are determined in this order as back, left, right, top, and bottom. Then, the virtual viewpoint video to which the shooting direction information has been added is output to the content generation unit 118.

(Embodiment 9)
In the fourth embodiment, an example has been described in which the storage unit 5 that stores digital content is incorporated in the image processing device 100. In the ninth embodiment, an example will be described in which digital content is stored in the external device 2102. Note that the image processing device 102 is a device obtained by removing the storage unit 5 from the image processing device 100 (not shown).

FIG. 21 is a diagram showing the system configuration of the image processing system 103 in the ninth embodiment. The image processing system 103 includes an image processing apparatus 102, a user device 2101, and an external device 2102.

The image processing device 102 generates digital content generated by the method described in any of the first to third embodiments. The generated digital content, media data such as virtual viewpoint images used for generation, icons representing each virtual viewpoint image, metadata of each virtual viewpoint image, and the like are transmitted to the external device 2102. Further, a display image according to any one of the fourth to sixth embodiments is generated. The generated display image is transmitted to the user device 2101.

The user device 2101 may be, for example, a PC, a smartphone, or a tablet terminal with a touch panel (not shown). In this embodiment, a tablet terminal having a touch panel will be described as an example.

The external device 2102 stores digital content generated by the method described in any of the first to third embodiments. Note that, like the storage unit 5 in FIG. 1, in addition to digital content, it also stores virtual viewpoint images, camera images, icons corresponding to virtual viewpoint images displayed in each digital content, and the like. When specific digital content is requested from the image processing apparatus 102, the corresponding digital content is transmitted to the image processing apparatus 102. Note that not only digital content but also virtual viewpoint images, metadata of each virtual viewpoint image, and the like may be transmitted to the image processing apparatus 102.

FIG. 22 is a diagram showing the flow of data transmission in the ninth embodiment. In this embodiment, a flow will be described in which a display image is generated in response to an instruction from a user, and a virtual viewpoint image to be displayed is changed in response to an operation on the display image.

In S2201, the user device 2101 transmits an instruction to view digital content to the image processing apparatus 102 through input from the user. Note that this instruction includes information for specifying the digital content that the user wants to view. Specifically, it is information such as an NFT of digital content and an address stored in the external device 2102.

In S2202, the image processing apparatus 102 requests the external device 2102 for digital content to be viewed based on the acquired viewing instruction.

In S2203, the external device 2102 identifies digital content corresponding to the obtained request. Note that not only the digital content but also metadata of the digital content and related virtual viewpoint images are specified according to the request content.

In S2204, the external device 2102 transmits the identified digital content to the image processing device 102. In addition to the digital content, if there is any specified information, it will also be sent.

In S2205, the image processing device 102 generates a display image corresponding to the acquired digital content. For example, if three virtual viewpoint videos are associated with the acquired digital content, a display image shown in FIG. 9 of the fourth embodiment is created. Note that the display image shown in FIG. 11 of the fifth embodiment may be created, or the display image shown in FIG. 13 of the sixth embodiment may be created. In this embodiment, it is assumed that the display image to be generated is set in advance to be the display image shown in FIG. 9 of the fourth embodiment. Note that a display image corresponding to the digital content may be set in advance. In that case, the creator sets it when creating the digital content and stores it in the metadata of the digital content. Furthermore, as another example, the user may instruct a display image to be generated. In that case, when instructing to view digital content in S2201, the type of display image to be displayed is also instructed. Each aspect of the acquired digital content is then associated with the generated display image. In this embodiment, the subject is a basketball player, and an image corresponding to a main image showing the player is displayed in the first display area 901. In the second display area 902, an image showing three virtual viewpoint videos related to the player displayed in the main image and an icon 913 showing the virtual viewpoint video are displayed in a superimposed manner. An image showing information about the team to which the player displayed in the main image belongs is displayed in the third display area 903. In the fourth display area 904, an image showing performance information of the player displayed in the main image in the season in which the main image was captured is displayed. An image showing the final score of the match at the time of shooting is displayed in the fifth display area 905. An image showing copyright information of the digital content is displayed in the sixth display area 906. Note that the first display area 901 to the sixth display area 906 are images that can be selected by user operation. In S2205, the main image showing the player displayed in the first display area 901 is displayed in the seventh display area 907 as an initial image.

In S2206, the image processing apparatus 102 transmits the display image generated in S2205 to the user device 2101.

In S2207, the user device 2101 displays the received display image.

In S2208, when the user device 2101 receives a user operation to select a display area of a display image, the user device 2101 transmits information specifying the selected display area to the image processing apparatus 102. For example, when the display image shown in FIG. 9 of the fourth embodiment is displayed and the user selects the image corresponding to the display area 902, information indicating that the display area 902 has been selected is transmitted to the image processing apparatus 102. Note that in this embodiment, information specifying the display area is transmitted to the image processing apparatus 102. This is so that an image or video different from the images displayed in the first display area 901 to the sixth display area 906 can be displayed in the seventh display area 907. Note that the present invention is not limited to this, and an image selected by a user operation may be displayed in the seventh display area 907. In that case, information indicating the selected image is sent to the image processing device 102 instead of information indicating the selected display area.

In S2209, the image processing apparatus 102 determines which image of the digital content has been selected from the icon corresponding to the selected display section. In S2208, since the display area 902 is selected, the virtual viewpoint video corresponding to the display area 902 is selected. Therefore, the virtual viewpoint video included in the digital content is displayed in the seventh display area 907. Note that if the digital content includes a plurality of virtual viewpoint videos, the virtual viewpoint video to be displayed first is set in advance. Through the above processing, information specifying the display area corresponding to the image selected by the user operation among the first display area 901 to the sixth display area 906 is received, and the image or video corresponding to the selected display area is displayed in the first display area 901 to the sixth display area 906. The display screen is updated by displaying it in the 7 display area 907. In addition, when a plurality of virtual viewpoint videos are displayed in the second area 1308 as in the display screen shown in FIG. 13 of the sixth embodiment, it is determined whether a touch input or flick input for the displayed virtual viewpoint videos has been received. do. When a touch input is received, the virtual viewpoint video being played is paused. When a flick input is received, the virtual viewpoint video displayed in the fifth display area 1305 and currently being played is switched to a different virtual viewpoint video.

In S2210, the image processing apparatus 102 transmits the updated display image to the user device 2101.

Through the above processing, a display image displaying digital content desired by the user can be created and displayed on the user device.

Although the present disclosure has been described in detail based on a plurality of embodiments, the present disclosure is not limited to the above embodiments, and various modifications can be made based on the gist of the present disclosure. It is not excluded from the scope of For example, the above embodiments 1 to 7 may be combined as appropriate.

Note that a computer program that implements some or all of the functions of the above-described embodiments of control in this embodiment may be supplied to an image processing system or the like via a network or various storage media. Then, a computer (or CPU, MPU, etc.) in the image processing system or the like may read and execute the program. In that case, the program and the storage medium storing the program constitute the present disclosure.

Note that the disclosure of this embodiment includes the following configuration, method, and program.

(Configuration 1) A virtual viewpoint image associated with the first surface of three-dimensional digital content, which is generated based on a plurality of images obtained by capturing images with a plurality of imaging devices and a virtual viewpoint. identifying means for identifying a virtual viewpoint image associated with a second surface of the digital content, and an image having a different viewpoint from the virtual viewpoint corresponding to the virtual viewpoint image associated with a second surface of the digital content;
Display control means for controlling display of an image corresponding to the virtual viewpoint image and an image corresponding to an image from a viewpoint different from the virtual viewpoint in a display area;
A device characterized by having:

(Configuration 2) further comprising an acquisition means for acquiring input information for selecting the display area,
The display control means displays an image corresponding to the selected display area in a selected image display area that is a different display area from the selected display area, based on the input information acquired by the acquisition means. The device according to configuration 1, characterized in that:

(Structure 3) The device according to Structure 1 or 2, wherein the digital content includes a plurality of virtual viewpoint images generated based on the plurality of images and a plurality of virtual viewpoints including the virtual viewpoint. .

(Structure 4) The apparatus according to any one of Structures 1 to 3, wherein the display control means displays images corresponding to the plurality of virtual viewpoint images in different display areas.

(Configuration 5) Associating images corresponding to the plurality of virtual viewpoint images with a specific display area,
The display control means selects a specific virtual viewpoint image from among the plurality of virtual viewpoint images when the acquisition means obtains an input for selecting a display area in which an image corresponding to the plurality of virtual viewpoint images is displayed. 5. The device according to any one of configurations 1 to 4, wherein the device displays the selected image in the selected image display area.

(Configuration 6) At least one virtual viewpoint among the plurality of virtual viewpoints corresponding to the plurality of virtual viewpoint images is included in at least one image among the plurality of images obtained by imaging with the plurality of imaging devices. 6. The apparatus according to any one of configurations 1 to 5, wherein the determination is made based on a subject to be photographed.

(Configuration 7) The device according to any one of configurations 1 to 6, wherein the position of the virtual viewpoint is determined based on the position of a three-dimensional shape representing the subject.

(Configuration 8) The position of at least one virtual viewpoint among the plurality of virtual viewpoints corresponding to the plurality of virtual viewpoint images is determined based on the position of the subject, and the line of sight direction from the virtual viewpoint is determined based on the orientation of the subject. 8. The device according to any one of configurations 1 to 7, characterized in that:

(Configuration 9) The position of at least one virtual viewpoint among the plurality of virtual viewpoints corresponding to the plurality of virtual viewpoint images is determined based on a position a predetermined distance behind the subject, and the line of sight direction from the virtual viewpoint is 8. The apparatus according to any one of configurations 1 to 7, wherein the determination is made based on the orientation of the subject.

(Configuration 10) The position of at least one virtual viewpoint among the plurality of virtual viewpoints corresponding to the plurality of virtual viewpoint images is determined based on the position on a spherical surface centered on the subject, and the line of sight direction from the virtual viewpoint is 8. The apparatus according to any one of configurations 1 to 7, wherein the determination is made based on a direction toward the subject from the position of the virtual viewpoint.

(Configuration 11) The subject is a person,
9. The device according to any one of configurations 1 to 8, wherein the orientation of the subject is the orientation of the face of the subject.

(Configuration 12) When specific operation information is input to the selected image display area, the display control means selects one of the virtual viewpoint images displayed in the selected image display area and the plurality of virtual viewpoint images. 6. The device according to any one of configurations 1 to 5, wherein the device switches between a virtual viewpoint image displayed in the selected image display area and a different virtual viewpoint image.

(Configuration 13) The specific operation information includes keyboard typing operation, mouse click operation, mouse scroll operation, touch operation on the display device on which the virtual viewpoint image is displayed, slide operation, flick gesture, pinch-in/pinch-out The device according to configuration 12, wherein the device is operation information regarding at least one of the operations.

(Configuration 14) The display control means displays icons corresponding to each of the plurality of virtual viewpoint images in a superimposed manner on the selected image display area, and upon receiving an input to select the icon, displays the icons on the selected image display area. 13. The device according to configuration 12, wherein the device switches between a virtual viewpoint image corresponding to the inputted icon and a virtual viewpoint image corresponding to the inputted icon.

(Structure 15) The apparatus according to any one of Structures 1 to 14, wherein the display control means displays an icon representing the virtual viewpoint image in a superimposed manner on an image representing the virtual viewpoint image.

(Method) A virtual viewpoint image associated with the first surface of three-dimensional digital content, which is generated based on a plurality of images obtained by capturing images with a plurality of imaging devices and a virtual viewpoint. identifying a virtual viewpoint image and an image with a different viewpoint from the virtual viewpoint corresponding to the virtual viewpoint image associated with the second side of the digital content;
a display control step of controlling display of an image corresponding to the virtual viewpoint image and an image corresponding to an image of a viewpoint different from the virtual viewpoint in a display area;
An image processing method comprising:

(Program) A program for controlling each means described in the apparatus described in any one of Configurations 1 to 15 by a computer.

1 Camera 2 Shape estimation section 3 Image generation section 4 Content generation section 5 Storage section 115 Display section 116 Operation section 100 Image processing device

Claims (17)

A virtual viewpoint image associated with the first surface of three-dimensional digital content, which is generated based on a plurality of images obtained by capturing images with a plurality of imaging devices and a virtual viewpoint. and identifying means for identifying an image of a different viewpoint from the virtual viewpoint corresponding to the virtual viewpoint image associated with the second side of the digital content;
Display control means for controlling display of an image corresponding to the virtual viewpoint image and an image corresponding to an image from a viewpoint different from the virtual viewpoint in a display area;
An image processing system comprising: Furthermore, it has an acquisition means for acquiring input information for selecting the display area,
The display control means displays an image corresponding to the selected display area in a selected image display area that is a different display area from the selected display area, based on the input information acquired by the acquisition means. The image processing system according to claim 1, characterized in that: The image processing system according to claim 2, wherein the digital content includes a plurality of virtual viewpoint images generated based on the plurality of images and a plurality of virtual viewpoints including the virtual viewpoint. 4. The image processing system according to claim 3, wherein the display control means displays images corresponding to the plurality of virtual viewpoint images in different display areas. associating images corresponding to the plurality of virtual viewpoint images with a specific display area;
The display control means selects a specific virtual viewpoint image from among the plurality of virtual viewpoint images when the acquisition means obtains an input for selecting a display area in which an image corresponding to the plurality of virtual viewpoint images is displayed. 4. The image processing system according to claim 3, wherein the image processing system displays the selected image in a selected image display area. At least one virtual viewpoint among the plurality of virtual viewpoints corresponding to the plurality of virtual viewpoint images is based on a subject included in at least one image among the plurality of images obtained by imaging with the plurality of imaging devices. The image processing system according to claim 5, wherein the image processing system is determined by: 7. The image processing system according to claim 6, wherein the position of the virtual viewpoint is determined based on the position of a three-dimensional shape representing the subject. The position of at least one virtual viewpoint among the plurality of virtual viewpoints corresponding to the plurality of virtual viewpoint images is determined based on the position of the subject, and the line of sight direction from the virtual viewpoint is determined based on the direction of the subject. The image processing system according to claim 3, characterized in that: The position of at least one virtual viewpoint among the plurality of virtual viewpoints corresponding to the plurality of virtual viewpoint images is determined based on a position a predetermined distance behind the subject, and the viewing direction from the virtual viewpoint is determined based on the direction of the subject. The image processing system according to claim 3, wherein the image processing system is determined based on. The position of at least one virtual viewpoint among the plurality of virtual viewpoints corresponding to the plurality of virtual viewpoint images is determined based on the position on a spherical surface centered on the subject, and the line of sight direction from the virtual viewpoint is determined based on the position on the spherical surface centered on the subject. The image processing system according to claim 3, wherein the image processing system is determined based on a direction from the position toward the subject. The subject is a person,
9. The image processing system according to claim 8, wherein the orientation of the subject is the orientation of the face of the subject. When specific operation information is input to the selected image display area, the display control means displays the virtual viewpoint image displayed in the selected image display area and the selected image from among the plurality of virtual viewpoint images. 6. The image processing system according to claim 5, wherein the virtual viewpoint image displayed in the area is switched between a different virtual viewpoint image. The specific operation information includes at least one of a keyboard typing operation, a mouse click operation, a scroll operation using a mouse, a touch operation on a display device on which a virtual viewpoint image is displayed, a slide operation, a flick gesture, and a pinch-in/pinch-out operation. The image processing system according to claim 12, wherein the image processing system is operation information regarding one of the above. The display control means superimposes and displays an icon corresponding to each of the plurality of virtual viewpoint images in the selected image display area, and upon receiving an input for selecting the icon, displays the icon corresponding to each of the plurality of virtual viewpoint images in the selected image display area. The image processing system according to claim 12, wherein an image and the virtual viewpoint image corresponding to the inputted icon are switched. The image processing system according to claim 1, wherein the display control means displays an icon representing the virtual viewpoint image in a superimposed manner on the image representing the virtual viewpoint image. A virtual viewpoint image associated with the first surface of three-dimensional digital content, which is generated based on a plurality of images obtained by capturing images with a plurality of imaging devices and a virtual viewpoint. and a specifying step of identifying an image of a different viewpoint from the virtual viewpoint corresponding to the virtual viewpoint image associated with the second side of the digital content;
a display control step of controlling display of an image corresponding to the virtual viewpoint image and an image corresponding to an image of a viewpoint different from the virtual viewpoint in a display area;
An image processing method comprising: A computer program for controlling each means of the image processing system according to any one of claims 1 to 15 by a computer.

Images