WO2019208702A1 - Free viewpoint image generation method, free viewpoint image display method, free viewpoint image generation device, and display device - Google Patents
Free viewpoint image generation method, free viewpoint image display method, free viewpoint image generation device, and display device Download PDFInfo
- Publication number
- WO2019208702A1 WO2019208702A1 PCT/JP2019/017658 JP2019017658W WO2019208702A1 WO 2019208702 A1 WO2019208702 A1 WO 2019208702A1 JP 2019017658 W JP2019017658 W JP 2019017658W WO 2019208702 A1 WO2019208702 A1 WO 2019208702A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- viewpoint image
- free viewpoint
- time
- display device
- image generation
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/194—Transmission of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
- H04N13/279—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals the virtual viewpoint locations being selected by the viewers or determined by tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2668—Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
Definitions
- the present disclosure relates to a free viewpoint image generation method, a free viewpoint image display method, a free viewpoint image generation device, and a display device.
- Patent Document 1 discloses a technique for distributing video shot from a plurality of viewpoints in conjunction with viewpoint movement.
- a 3D model of the scene is generated using a group of images obtained by photographing a specific scene with a plurality of calibrated cameras, and a free viewpoint image (free image) of the scene viewed from a free viewpoint using the 3D model.
- a technique for generating (viewpoint video) is known.
- This disclosure is intended to provide a free viewpoint image generation method, a free viewpoint image display method, a free viewpoint image generation apparatus, or a display apparatus that allows a user to easily select a target for generating a free viewpoint image.
- a free viewpoint image generation method includes location information indicating a first place where a free viewpoint image is generated and a first time period indicating a first time zone where the free viewpoint image is generated. Information is received from the display device, and the first location is transmitted to the display device in the first time taken in the first time zone, and the display device is selected using the first video. , Receiving specified time information indicating a specified time included in the first time zone from the display device, generating the free viewpoint image at the specified time of the first place, and displaying the free viewpoint image on the display device Send to.
- a free viewpoint image display method includes: location information indicating a first place where a free viewpoint image is to be generated; and a first time indicating a first time zone where the free viewpoint image is to be generated.
- Information is transmitted to a free viewpoint image generation device, a first video in which the first location is captured in the first time zone is received from the free viewpoint image generation device, and the first video is used to A first user interface for selecting a designated time included in one hour zone is displayed, and designated time information indicating the designated time selected via the first user interface is displayed in the free viewpoint image generation device. Transmitting, receiving the free viewpoint image at the specified time of the first place from the free viewpoint image generating device, and displaying the free viewpoint image.
- This disclosure can provide a free viewpoint image generation method, a free viewpoint image display method, a free viewpoint image generation apparatus, or a display apparatus that allows a user to easily select a target for generating a free viewpoint image.
- FIG. 1 is a diagram illustrating an outline of a free viewpoint image generation system according to an embodiment.
- FIG. 2 is a diagram illustrating a configuration of the free viewpoint image generation system according to the embodiment.
- FIG. 3 is a block diagram of the free viewpoint image generation apparatus according to the embodiment.
- FIG. 4 is a block diagram of the display device according to the embodiment.
- FIG. 5 is a diagram illustrating an operation of the free viewpoint image generation system according to the embodiment.
- FIG. 6 is a flowchart of a start signal transmission process performed by the display device according to the embodiment.
- FIG. 7A is a diagram illustrating an example of a setting screen according to the embodiment.
- FIG. 7B is a diagram illustrating an example of a setting screen according to the embodiment.
- FIG. 8 is a flowchart of processing when a start signal is received by the free viewpoint image generation device according to the embodiment.
- FIG. 9 is a flowchart of camera video generation processing by the free viewpoint image generation device according to the embodiment.
- FIG. 10 is a flowchart of a designated time selection process performed by the display device according to the embodiment.
- FIG. 11 is a diagram illustrating an example of a distribution video display screen according to the embodiment.
- FIG. 12 is a flowchart of a 3D model generation process performed by the free viewpoint image generation apparatus according to the embodiment.
- FIG. 13 is a flowchart of viewpoint selection processing by the display device according to the embodiment.
- FIG. 14 is a diagram illustrating an example of a 3D model display screen according to the embodiment.
- FIG. 15 is a flowchart of free viewpoint image generation processing by the free viewpoint image generation apparatus according to the embodiment.
- FIG. 16 is a flowchart of free viewpoint image display processing by the display device according to the embodiment.
- FIG. 17 is a diagram illustrating an example of a free viewpoint image display screen according to the embodiment.
- FIG. 18 is a flowchart of processing by the free viewpoint image generation device according to the embodiment.
- FIG. 19 is a flowchart of processing performed by the display device according to the embodiment.
- a free viewpoint image generation method includes location information indicating a first place where a free viewpoint image is generated and a first time period indicating a first time zone where the free viewpoint image is generated. Information is received from the display device, and the first location is transmitted to the display device in the first time taken in the first time zone, and the display device is selected using the first video. , Receiving specified time information indicating a specified time included in the first time zone from the display device, generating the free viewpoint image at the specified time of the first place, and displaying the free viewpoint image on the display device Send to.
- the user of the display device can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
- the second time information indicating a second time zone including the first time zone is received from the display device, and the second time information is displayed.
- Thumbnail images at a plurality of times included in a time zone may be transmitted to the display device, and the first time zone may be specified using the thumbnail images on the display device.
- the user of the display device can designate a detailed time zone using the thumbnail image. Therefore, the user can easily select a target for generating a free viewpoint image.
- the free viewpoint image generation method further generates a three-dimensional model of the first place at the designated time, transmits the three-dimensional model to the display device, and the display device displays the three-dimensional model.
- the viewpoint information indicating the selected viewpoint selected by use may be received from the display device, and the free viewpoint image may be an image viewed from the selected viewpoint.
- the user of the display device can select the viewpoint of the free viewpoint image using the three-dimensional model. Therefore, the user can easily select the viewpoint of the free viewpoint image.
- a free viewpoint image display method includes: location information indicating a first place where a free viewpoint image is to be generated; and a first time indicating a first time zone where the free viewpoint image is to be generated.
- Information is transmitted to a free viewpoint image generation device, a first video in which the first location is captured in the first time zone is received from the free viewpoint image generation device, and the first video is used to A first user interface for selecting a designated time included in one hour zone is displayed, and designated time information indicating the designated time selected via the first user interface is displayed in the free viewpoint image generation device. Transmitting, receiving the free viewpoint image at the specified time of the first place from the free viewpoint image generating device, and displaying the free viewpoint image.
- the user of the display device can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
- second time information indicating a second time zone including the first time zone is transmitted to the free viewpoint image generation device.
- the first time zone may be specified via the second user interface.
- the user of the display device can designate a detailed time zone using the thumbnail image. Therefore, the user can easily select a target for generating a free viewpoint image.
- the free viewpoint image display method further receives a three-dimensional model at the specified time of the first place from the free viewpoint image generation device, and selects a selected viewpoint using the three-dimensional model.
- a third user interface is displayed, and viewpoint information indicating the selected viewpoint selected via the third user interface is transmitted to the free viewpoint image generation device, and the free viewpoint image is viewed from the selected viewpoint. It may be an image.
- the user of the display device can select the viewpoint of the free viewpoint image using the three-dimensional model. Therefore, the user can easily select the viewpoint of the free viewpoint image.
- a free viewpoint image generation device is a free viewpoint image generation device that generates a three-dimensional model from a plurality of images, and includes a processor and a memory, and the processor includes the memory. And receiving from the display device location information indicating a first location for generating a free viewpoint image and first time information indicating a first time zone for generating the free viewpoint image; A designated time included in the first time zone, which is selected using the first video in the display device by transmitting a first video imaged in one place in the first time zone to the display device. Is received from the display device, the free viewpoint image at the specified time at the first location is generated, and the free viewpoint image is transmitted to the display device.
- the user of the display device can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
- a display device includes a processor and a memory, and the processor uses the memory to store location information indicating a first location where a free viewpoint image is generated, and the free information.
- First time information indicating a first time zone for which a viewpoint image is to be generated is transmitted to a free viewpoint image generating device, and the first video in which the first place is captured in the first time zone is the free time.
- a first user interface received from the viewpoint image generation device and used to select a designated time included in the first time zone using the first video is displayed and selected via the first user interface.
- the designated time information indicating the designated time is transmitted to the free viewpoint image generating device, the free viewpoint image at the designated time of the first place is received from the free viewpoint image generating device, and the free view image is received. To display the image.
- the user of the display device can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
- a free viewpoint image generation system capable of easily realizing generation of a free viewpoint image using a multi-view camera and its video will be described.
- FIG. 1 is a diagram showing an outline of a free viewpoint image generation system.
- a 3D space by photographing the same space from multiple viewpoints using a calibrated camera (for example, a fixed camera) (3D space reconstruction).
- a calibrated camera for example, a fixed camera
- 3D space reconstruction By using this three-dimensional reconstructed data for tracking, scene analysis, and video rendering, a video viewed from an arbitrary viewpoint (free viewpoint camera) can be generated. Thereby, a next generation wide area monitoring system and a free viewpoint image generation system can be realized.
- the display device acquires a camera video including a time at which a free viewpoint image is to be generated by using the location and time zone information, and generates a free viewpoint time from the camera video. Specify the time you want to generate, and use the 3D model to specify the viewpoint from which you want to generate a free viewpoint image.
- the system which can acquire the free viewpoint image which a user asks easily can be realized.
- the user can carry out a complicated processing procedure from acquiring a free viewpoint image from a camera image by performing a series of processes, and thus it is possible to easily use the free viewpoint image. .
- FIG. 2 is a diagram illustrating a configuration example of the free viewpoint image generation system 100 according to the present embodiment.
- the free viewpoint image generation system 100 includes a plurality of cameras 101, a free viewpoint image generation apparatus 102, and a plurality of display apparatuses 103.
- the plurality of cameras 101 generate a plurality of camera images by photographing the same scene from different viewpoints.
- the plurality of cameras 101 may perform synchronous shooting.
- the plurality of cameras 101 may embed time information in the video or give index information indicating the frame order to the video.
- the free viewpoint image generation device 102 is a server that generates a free viewpoint image, acquires a plurality of camera images captured by a plurality of cameras 101, and uses the plurality of camera images to generate a three-dimensional model and a free viewpoint image. Is generated.
- the free viewpoint image generation device 102 is connected to a plurality of display devices 103 via the network 104.
- the free viewpoint image generation device 102 transmits the camera video, the three-dimensional model, and the free viewpoint image to the display device 103.
- the free viewpoint image generation device 102 generates a three-dimensional model and a free viewpoint image based on the time and viewpoint information transmitted from the display device 103, respectively. Transmit to the display device 103.
- the free viewpoint image generation device 102 may generate a free viewpoint image using a three-dimensional model, or may generate a free viewpoint image by interpolating images from two or more images. .
- the display device 103 is a terminal that presents to the user by displaying the camera video, the three-dimensional model, and the free viewpoint image transmitted from the free viewpoint image generating device 102. Further, the display device 103 selects time from the camera video, selects a viewpoint from the three-dimensional model, and transmits each information to the free viewpoint image generation device 102.
- the communication method is not particularly limited as long as data can be exchanged between the free viewpoint image generation device 102 and the display device 103.
- communication between remote locations may be performed via the Internet, or may be performed within a LAN in a laboratory or the like.
- FIG. 3 is a block diagram of the free viewpoint image generation apparatus 102 according to the present embodiment.
- the free viewpoint image generation apparatus 102 includes an overall setting unit 201, a camera video distribution unit 202, a 3D model generation unit 203, a free viewpoint image generation unit 204, a communication unit 205, a setting storage unit 206, and a camera video.
- a storage unit 207, a camera parameter storage unit 208, a 3D model storage unit 209, and a free viewpoint image storage unit 210 are provided.
- the overall setting unit 201 sets a setting time and a setting location, which are the time and place of a video used when generating a free viewpoint image, and stores information indicating the set setting time and the setting location in the setting storage unit 206. . For example, at the time of verification of an accident site, several minutes before and after reflecting the moment of occurrence of the accident are set as the set time, and the accident occurrence place is set as the set place. For example, the overall setting unit 201 sets the time and place designated from the display device 103 through the communication unit 205 as the setting time and setting place. Alternatively, the overall setting unit 201 automatically sets a setting location and a setting time by recognizing an accident using image processing such as detecting a crowd, and notifies the display device 103 of the set setting location and setting information. May be.
- the camera video distribution unit 202 acquires the camera video corresponding to the set time and the set location set by the overall setting unit 201 and transmits the acquired camera video to the display device 103.
- the 3D model generation unit 203 receives the time at which the 3D model is to be generated from the display device 103, and generates a 3D model at the corresponding time. In addition, the 3D model generation unit 203 transmits the generated 3D model to the display device 103 and stores the 3D model in the 3D model storage unit 209. For example, the 3D model generation unit 203 generates a 3D model by performing 3D reconstruction using an image processing technique such as SfM (Structure from Motion). Alternatively, the three-dimensional generation unit 203 generates a three-dimensional model using the depth information when the camera video includes depth information.
- SfM Structure from Motion
- the free viewpoint image generation unit 204 receives the viewpoint information transmitted from the display device 103, generates a free viewpoint image viewed from the viewpoint, and transmits the generated free viewpoint image to the display device 103.
- the free viewpoint image generation unit 204 stores the generated free viewpoint image in the free viewpoint image storage unit 210.
- the free viewpoint image generation unit 204 generates a free viewpoint image using a three-dimensional model.
- the free viewpoint image generation unit 204 generates a free viewpoint image by interpolating images between cameras two-dimensionally as in morphing processing.
- the free viewpoint image (free viewpoint video) may be a still image or a moving image.
- the moving image may be a moving image showing a time-series change of a specific scene viewed from a certain viewpoint, or may be a moving image in which the viewpoint position is continuously changed at a specified time. A combination of these may also be used.
- the communication unit 205 exchanges data with the display device 103 and the overall setting unit 201, the camera video distribution unit 202, the 3D model generation unit 203, and the free viewpoint image generation unit 204.
- the setting storage unit 206 stores information related to the entire process regarding the time zone and place where the free viewpoint image is to be generated.
- the camera video storage unit 207 stores the camera video shot by the camera 101. Specifically, the camera video storage unit 207 stores the camera video acquired directly from the camera 101. Alternatively, the camera video storage unit 207 may store a camera video acquired indirectly through another device.
- the camera parameter storage unit 208 stores camera parameters including the three-dimensional position and orientation information of the camera 101 that captured the camera video stored in the camera video storage unit 207.
- the three-dimensional position and orientation information is information acquired by, for example, a GPS or gyro sensor provided in the camera 101.
- the free viewpoint image generation apparatus 102 may estimate the three-dimensional position and orientation information using an image processing technique such as SfM based on the camera video.
- the 3D model storage unit 209 stores the 3D model generated by the 3D model generation unit 203.
- the free viewpoint image storage unit 210 stores the free viewpoint image generated by the free viewpoint image generation unit 204.
- FIG. 4 is a block diagram of the display device 103 according to the present embodiment.
- the display device 103 includes a setting screen generation unit 301, a distribution video display screen generation unit 302, a 3D model display screen generation unit 303, a free viewpoint image display screen generation unit 304, a communication unit 305, and a screen control unit 306.
- the setting screen generation unit 301 generates a setting screen that is an interface for generating a start signal including time zone information and location information for acquiring a video including a time for generating a free viewpoint image.
- the screen is not limited to a static image, and may include a moving image, or may include an icon, an operation unit, and the like whose display form changes with time.
- the distribution video display screen generation unit 302 generates a distribution video display screen that is an interface for selecting a time for generating a free viewpoint image from the acquired camera video.
- the 3D model display screen generation unit 303 generates a 3D model display screen that is an interface for determining a viewpoint for generating a free viewpoint image using the acquired 3D model.
- the free viewpoint image display screen generation unit 304 generates a free viewpoint image display screen that is a screen for displaying the acquired free viewpoint image.
- the communication unit 305 communicates with the free viewpoint image generation device 102 and is used by the setting screen generation unit 301, the distribution video display screen generation unit 302, the 3D model display screen generation unit 303, and the free viewpoint image display screen generation unit 304. Send and receive information.
- the screen control unit 306 includes any one of the display screens generated by the setting screen generation unit 301, the distribution video display screen generation unit 302, the 3D model display screen generation unit 303, and the free viewpoint image display screen generation unit 304. Or display on the screen display unit 307 and control the display timing.
- the screen control unit 306 acquires a user operation performed on the screen display unit 307, and the setting screen generation unit 301, the distribution video display screen generation unit 302, the 3D model display screen generation unit 303, and Information such as user operations is sent to the free viewpoint image display screen generation unit 304.
- the screen display unit 307 presents various information to the user by displaying the screen acquired from the screen control unit 306.
- the camera video storage unit 308 is a storage for storing the camera video transmitted from the free viewpoint image generation device 102.
- the 3D model storage unit 309 is a storage that stores the 3D model transmitted from the free viewpoint image generation apparatus 102.
- the free viewpoint image storage unit 310 is a storage for storing the free viewpoint image transmitted from the free viewpoint image generation apparatus 102.
- the camera image storage unit 308, the 3D model storage unit 309, and the free viewpoint image storage unit 310 only need to be able to store data temporarily and for a long period of time. ) That can hold data for a long period of time.
- FIG. 5 is a sequence diagram showing the operation of the free viewpoint image generation system 100 according to the present embodiment.
- FIG. 6 is a flowchart of the start signal transmission process (S101).
- the display apparatus 103 determines whether candidate information has been received from the free viewpoint image generation apparatus 102 (S211).
- the candidate information is, for example, information indicating a plurality of candidates for the location and time (time zone or time) of the target for generating the free viewpoint image.
- step S101A shown in FIG. 5 the display device 103 has not received candidate information (No in S211).
- the display device 103 displays a setting screen 401A that does not include candidate information.
- FIG. 7A is a diagram showing an example of the setting screen 401A.
- the setting screen 401A includes a time input unit 411 for inputting time information and a place input unit 412 for inputting place information.
- the setting screen 401A shown here is merely an example, and the display device 103 only needs to have an interface through which input information for specifying a time and place for generating a free viewpoint image can be input.
- FIG. 7A shows an example in which time information and location information are input as an example of input information, but it is sufficient if the user can select a video including a time for which a free viewpoint image is to be generated. May be used. Further, any method may be used as the input method, for example, voice input or the like may be used.
- step S101B shown in FIG. 5 the display apparatus 103 receives candidate information (Yes in S211).
- the display device 103 generates a setting screen 401B including the received candidate information (S212), and displays the generated setting screen 401B (S213).
- FIG. 7B is a diagram showing an example of the setting screen 401B.
- the setting screen 401B includes a candidate information display unit 413 that displays candidate information in addition to the contents of the setting screen 401A.
- the candidate information is, for example, a thumbnail image at a certain time (for example, every 5 minutes) of the video at the same point. Thereby, since the user can estimate the time when the accident or the like has occurred, the user can easily select the time. Note that the time corresponding to the thumbnail image may be automatically selected by the user selecting the thumbnail image.
- the display device 103 acquires time information indicating the time input by the user using the setting screen 401A or 401B (S214). For example, the user inputs time information to the time input unit 411. Note that this time information may be input in units of seconds or rough input in units of time. That is, the time information may indicate a time or a time zone.
- the display device 103 acquires location information input by the user using the setting screen 401A or 401B (S215).
- the user inputs location information to the location input unit 412.
- the location information may be an address, a unique name of an intersection or a building, a latitude / longitude, or an ID (identifier) arbitrarily set by the user. May be.
- the location information may indicate a rough location such as a bay area or an industrial area. For example, in step S101A shown in FIG. 5, a rough time is specified, and a detailed time is specified in step S101B.
- the display apparatus 103 transmits a start signal including the time information acquired in step S214 and the location information acquired in step S215 to the free viewpoint image generation apparatus 102 (S216).
- FIG. 8 is a flowchart showing the operation (S102) of the overall setting unit 201 of the free viewpoint image generation apparatus 102 that has received the start signal.
- the overall setting unit 201 determines whether a start signal has been received from the display device 103 (S221). If the start signal is not received (No in S221), the overall setting unit 201 performs Step S221 again. That is, the entire setting unit 201 continues to wait until it receives a start signal.
- the overall setting unit 201 determines whether information (location information and time information) included in the received start signal is unique (S222).
- the information is unique indicates a state in which a video to be transmitted to the display device 103 can be uniquely determined based on location information and time information.
- this state is a case where the image is specified like an image of the intersection A at 13:00.
- the overall setting unit 201 transmits a distribution start signal for starting distribution of the video uniquely determined by the information included in the start signal to the camera video distribution unit 202. (S223). For example, in step S ⁇ b> 102 ⁇ / b> B illustrated in FIG. 5, the information is unique, and the overall setting unit 201 transmits a distribution start signal to the camera video distribution unit 202.
- the overall setting unit 201 generates candidate information by acquiring or generating information necessary to clarify the ambiguous information included in the start signal (S224). ).
- the ambiguous information here refers to a case where the time information includes a range of a certain time or more, such as around noon or in the morning, instead of 13:00.
- the candidate information at that time there is a thumbnail image group in about 10 minutes from 11:00 to 13:00 around noon.
- the overall setting unit 201 transmits the generated candidate information to the display device 103 (S225).
- the whole setting part 201 can assist the operation
- the information is not unique, and the entire setting unit 201 transmits candidate information to the display device 103.
- a narrowing operation is performed.
- the display device 103 acquires, for example, thumbnail images in units of 5 minutes and displays them as a list.
- the user can search for accurate time through a user interface capable of performing the narrowing down operation, and thus can search for accurate time information.
- time information is not unique
- the display device 103 displays thumbnail images of a plurality of points included in the area at the time designated by the time information. The user can select detailed location information by selecting a desired location from the displayed thumbnail images.
- time information and location information included in the database may be used in cooperation with a database that summarizes accident information.
- FIG. 9 is a flowchart showing the operation (S103) of the camera video distribution unit 202 that has received the distribution start signal.
- the camera video distribution unit 202 determines whether a distribution start signal has been received (S231). When the distribution start signal has not been received (No in S231), the camera video distribution unit 202 performs Step S231 again. That is, the camera video distribution unit 202 continues to wait until a distribution start signal is received.
- the camera video distribution unit 202 When receiving the distribution start signal (Yes in S231), the camera video distribution unit 202 acquires the time information and the location information included in the distribution start signal, and determines the location and time related to the video to be distributed (S232). For example, the camera video distribution unit 202 determines the location and time indicated by the location information and time information as the location and time of the video to be distributed.
- the camera video distribution unit 202 acquires the video corresponding to the location and time determined in step S232 from the camera video storage unit 207 (S233). Note that the camera video distribution unit 202 may directly acquire the camera video from the camera 101 when the camera video can be acquired directly from the camera 101, such as when the current time is designated. Finally, the camera video distribution unit 202 transmits the acquired camera video to the display device 103 (S234).
- FIG. 10 is a flowchart showing the process (S104) in the display device 103 that has received the camera video.
- the display device 103 determines whether a camera video has been received from the free viewpoint image generation device 102 (S241). If the display device 103 has not received the camera video (No in S241), the display device 103 performs Step S241 again. That is, the display device 103 continues to wait until a camera video is received.
- the display device 103 When the display device 103 receives the camera video (Yes in S241), the display device 103 generates the distribution video display screen 402 using the received camera video (S242). Next, the display device 103 presents the generated distribution video display screen 402 to the user using the screen control unit 306 and the screen display unit 307 (S243).
- FIG. 11 is a diagram illustrating an example of the distribution video display screen 402.
- the distribution video display screen 402 includes a video display unit 421 on which the received camera video is displayed, and an operation unit 422 for designating a time at which a 3D model is to be generated.
- the display device 103 may receive a plurality of camera videos and display a plurality of camera videos on the video display unit 421.
- the plurality of camera videos are, for example, camera videos obtained by shooting the same scene from different viewpoints in the same time zone.
- the display device 103 may display all of the plurality of camera images on the image display unit 421 at the same time, or may select and display one or more viewpoint images from among the plurality of camera images.
- an interface may be provided so that the user can switch the video.
- the operation unit 422 may have a function of playing and stopping video and a function of determining a designated time that is a time for generating a three-dimensional model. In order to facilitate the determination of the designated time by the user, the operation unit 422 may have a function of visualizing the time display in the video using a slide bar or a time code. Further, the operation unit 422 may have a function of changing the playback speed or playback direction such as fast forward or rewind.
- the user inputs the designated time by adjusting the slide bar of the operation unit 422 to the designated time and performing a decision operation (for example, a decision button operation).
- a decision operation for example, a decision button operation.
- the method for inputting the designated time is not limited to this, and any method may be used.
- the display device 103 acquires the designated time input by the user via the distribution video display screen 402 (S244). Next, the display device 103 transmits specified time information indicating the acquired specified time to the free viewpoint image generation device 102 (S245).
- FIG. 12 is a flowchart of the process (S105) by the three-dimensional model generation unit 203 of the free viewpoint image generation apparatus 102 that has received the specified time information.
- the three-dimensional model generation unit 203 determines whether designated time information has been received from the display device 103 (S251). If the specified time information has not been received (No in S251), the three-dimensional model generation unit 203 performs Step S251 again. That is, the three-dimensional model generation unit 203 continues to wait until receiving specified time information.
- 3D model generation unit 203 receives the specified time information (Yes in S251), and generates a 3D model at the specified time indicated by the specified time information (S252). For example, the 3D model generation unit 203 generates a 3D model using PMVS (Patch-based Multi View Stereo). Specifically, the three-dimensional model generation unit 203 determines the same feature point between images based on feature points in the image such as a corner of an object, and the camera feature is determined for the determined feature points. A 3D model is generated by performing a 3D survey using camera parameters including a 3D posture. Alternatively, the three-dimensional model generation unit 203 may generate a three-dimensional model using a visual volume intersection method. That is, the three-dimensional model generation unit 203 may generate a three-dimensional model by integrating the object regions captured by each camera in three dimensions.
- PMVS Packe-based Multi View Stereo
- the 3D model generation unit 203 transmits the generated 3D model to the display device 103 (S253).
- any method may be used for transmitting information related to the three-dimensional model to the display device 103.
- the three-dimensional model generation unit 203 may once write the three-dimensional model in some file such as text and transmit the file.
- the 3D model generation unit 203 may appropriately stream the generated 3D model data. In the former, it becomes easy to manage a three-dimensional model for each time in the free viewpoint image generation apparatus 102 and the display apparatus 103. In the latter method, the time required for transmission to the display device 103 can be shortened.
- the 3D model generation unit 203 may start generating a 3D model at a time before and after the specified time after generating a 3D model at the specified time. For example, when the designated time is time n, the 3D model generation unit 203 automatically generates a 3D model at time n + 1 and time n ⁇ 1 after generating and transmitting the 3D model at time n. May be. Further, the three-dimensional model generation unit 203 may or may not transmit the generated three-dimensional model at time n + 1 and time n ⁇ 1 to the display device 103. According to this, a three-dimensional model is created in advance such as several frames before and after the designated time, specifically, before and after the moment of a traffic accident. The user has a high possibility of seeing a free viewpoint image at a time around the designated time. Therefore, by making preparations in advance in this way, the waiting time for generating the three-dimensional model is unnecessary, and the waiting time of the user can be reduced.
- the three-dimensional model generation unit 203 generates a three-dimensional model at the designated time and the times before and after it in step S252, and transmits the generated three-dimensional model at a plurality of times to the display device 103 in step S253.
- multiple times or time zones are specified in the specified time information, and the 3D model generation unit 203 generates a 3D model of the specified multiple times or time zones and transmits the generated 3D model to the display device 103. May be.
- FIG. 13 is a flowchart showing the process (S106) in the display device 103 that has received the three-dimensional model.
- the display apparatus 103 determines whether a three-dimensional model has been received from the free viewpoint image generation apparatus 102 (S261). When the display device 103 has not received the three-dimensional model (No in S261), the display device 103 performs Step S261 again. That is, the display device 103 continues to wait until a three-dimensional model is received.
- the display device 103 When receiving the 3D model (Yes in S261), the display device 103 generates the 3D model display screen 403 using the received 3D model (S262). Next, the display device 103 presents the generated 3D model display screen 403 to the user using the screen control unit 306 and the screen display unit 307 (S263).
- FIG. 14 is a diagram illustrating an example of the three-dimensional model display screen 403.
- the 3D model display screen 403 includes a 3D model display unit 431 for displaying the received 3D model, a display operation unit 432 for operating the 3D model displayed on the 3D model display unit 431, A viewpoint determination unit 433 is included to determine the viewpoint of the three-dimensional mole currently displayed as viewpoint information to be sent to the free viewpoint image generation apparatus 102.
- the 3D model display unit 431 displays a 2D image obtained by viewing the 3D model from a predetermined viewpoint.
- the display device 103 sets the initial viewpoint to be displayed to the same viewpoint as the camera video displayed on the distribution video display screen 402.
- the display device 103 may analyze the coordinate information in the three-dimensional model and set the initial viewpoint so that the object of interest (for example, the largest object) can be seen well.
- the display device 103 may set the initial viewpoint so that the object of interest is displayed in the middle of the image.
- the display operation unit 432 has a function for the user to perform an operation of changing the viewpoint (virtual camera position) of the image displayed on the 3D model display unit 431. Specifically, it is only necessary to change the position and size of the three-dimensional model as expected by the user by controlling the movement and rotation of the viewpoint and the gazing point. That is, the interface is not particularly limited, and the movement of the viewpoint may be controlled by a keyboard, or an interface such as a button may be prepared as shown in FIG. Alternatively, the viewpoint may be controlled by touching the 3D model display unit 431.
- the viewpoint determination unit 433 is an operation unit for determining a viewpoint capable of reproducing the appearance state of the 3D model currently displayed on the 3D model display unit 431 as a viewpoint to be transmitted to the free viewpoint image generation apparatus 102. .
- the viewpoint determination unit 433 may be included in the three-dimensional model display unit 431 or the display operation unit 432. That is, the viewpoint may be determined by the user performing a predetermined operation on the 3D model display unit 431 or the display operation unit 432.
- the display device 103 acquires viewpoint information indicating a viewpoint designated by a user operation on the three-dimensional model display screen 403 (S264). Finally, the display apparatus 103 transmits the acquired viewpoint information to the free viewpoint image generation apparatus 102 (S265).
- FIG. 15 is a flowchart showing the processing (S107) of the free viewpoint image generation unit 204 of the free viewpoint image generation apparatus 102 that has received the viewpoint information.
- the free viewpoint image generation unit 204 determines whether or not viewpoint information for generating a free viewpoint image has been received from the display device 103 (S271). When the viewpoint information has not been received (No in S271), the free viewpoint image generation unit 204 performs Step S271 again. That is, the free viewpoint image generation unit 204 continues to wait until it receives viewpoint information.
- the free viewpoint image generation unit 204 When the viewpoint information is received (Yes in S271), the free viewpoint image generation unit 204 generates a free viewpoint image at a specified time viewed from the viewpoint indicated by the received viewpoint information (S272). For example, the free viewpoint image generation unit 204 assumes a three-dimensional space using a three-dimensional model, generates a free viewpoint image by simulating and rendering the appearance from a specified viewpoint. Alternatively, the free viewpoint image generation unit 204 may generate a free viewpoint image by interpolating two or three camera images close to the designated viewpoint.
- the free viewpoint image generation unit 204 transmits the generated free viewpoint image to the display device 103 (S273).
- the free viewpoint image generation unit 204 may generate a free viewpoint image when viewed from the surrounding viewpoints after generating a free viewpoint image based on the viewpoint information. Specifically, the free viewpoint image generation unit 204 may analyze the three-dimensional model and generate a free viewpoint image of a viewpoint that faces the object rather than the designated viewpoint.
- the free viewpoint image generation unit 204 may generate a free viewpoint image viewed from the viewpoint indicated by the viewpoint information at times before and after the specified time specified by the specified time information. According to this, the user can confirm the time change at the designated viewpoint as an image, and can effectively present information to the user.
- the free viewpoint image generation unit 204 may generate a free viewpoint image having a field angle slightly wider than the designated viewpoint.
- the designated viewpoint can be reproduced by matching the display range with the designated viewpoint in the display device 103. According to this, in the display device 103, when the designated viewpoint is narrower than expected, the user can visually recognize the peripheral area without performing the free viewpoint image generation process again.
- the free viewpoint image generation unit 204 may generate a more telephoto image at the same position and in the same line-of-sight direction in addition to generating the free viewpoint image of the designated viewpoint. . According to this, the free viewpoint image generation unit 204 transmits the telephoto image together with the designated viewpoint to the display device 103 so that when the zoom operation is performed on the display device 103, these images are displayed while being switched. Can do. This makes it possible to present a high-definition image to the user even after zooming.
- FIG. 16 is a flowchart showing the processing (S108) of the display device 103 that has received the free viewpoint image.
- the display apparatus 103 determines whether a free viewpoint image has been received from the free viewpoint image generation apparatus 102 (S281). When the display device 103 has not received the free viewpoint image (No in S281), the display device 103 performs Step S281 again. That is, the display device 103 continues to wait until a free viewpoint image is received.
- the display device 103 When the display device 103 receives the free viewpoint image (Yes in S281), the display device 103 generates the free viewpoint image display screen 404 using the received free viewpoint image (S282). Next, the display apparatus 103 presents the generated free viewpoint image display screen 404 to the user using the screen control unit 306 and the screen display unit 307 (S283).
- FIG. 17 is a diagram illustrating an example of the free viewpoint image display screen 404.
- the free viewpoint image display screen 404 includes a display unit 441 that displays at least the acquired free viewpoint image.
- the free viewpoint image display screen 404 may include a history unit 442 that displays thumbnail images and the like of previously generated free viewpoint images.
- the display unit 441 displays the free viewpoint image received from the free viewpoint image generation device 102.
- the plurality of free viewpoint images may be displayed as free viewpoint images.
- operations such as display enlargement / reduction and rotation may be performed. This operation may be performed by an interface such as a touch operation, or may be performed by an operation using a keyboard and a mouse.
- the free viewpoint image display screen 404 includes an interface such as a button, and an operation may be performed using the interface.
- the history unit 442 is used when a free viewpoint image generated previously is viewed later.
- the display device 103 stores the acquired free viewpoint image in the free viewpoint image storage unit 310 every time a free viewpoint image is acquired.
- the display device 103 can acquire a past free viewpoint image from the display device 103 when displaying the history content, so that the process of generating the free viewpoint image again becomes unnecessary. Therefore, reduction in processing time can be realized.
- the free viewpoint image generation apparatus 102 performs the processing shown in FIG. First, the free viewpoint image generation device 102 displays location information indicating a first location where a free viewpoint image is to be generated and first time information indicating a first time zone where the free viewpoint image is to be generated. 103 (S301: S102B in FIG. 5, S221 in FIG. 8). Next, the free viewpoint image generation apparatus 102 transmits the first video (camera video) taken at the first place in the first time zone to the display device 103 (S302: S103 in FIG. 5, S103 in FIG. 9). S234).
- the first video camera video
- the free viewpoint image generation device 102 receives from the display device 103 specified time information indicating a specified time included in the first time zone, which is selected using the first video on the display device 103 (S303: FIG. 5 S105, FIG. 12 S251). Next, the free viewpoint image generation device 102 generates a free viewpoint image at the designated time of the first place (S304: S107 in FIG. 5 and S272 in FIG. 15), and transmits the free viewpoint image to the display device (S305). : S107 in FIG. 5 and S273 in FIG. 15).
- the user of the display device 103 can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
- the free viewpoint image generation device 102 receives the second time information indicating the second time zone including the first time zone from the display device 103 before receiving the first time information from the display device 103. (S102A in FIG. 5 and S221 in FIG. 8). Next, the free viewpoint image generation device 102 transmits thumbnail images (candidate information) at a plurality of times included in the second time zone to the display device 103 (S102A in FIG. 5 and S225 in FIG. 8). The first time zone is designated using a thumbnail image on the display device 103.
- the user of the display device 103 can designate a detailed time zone using the thumbnail image. Therefore, the user can easily select a target for generating a free viewpoint image.
- the free viewpoint image generation apparatus 102 further generates a three-dimensional model at the designated time of the first place (S105 in FIG. 5 and S252 in FIG. 12), and transmits the three-dimensional model to the display apparatus 103 (FIG. 5 (S105, S253 in FIG. 12), the viewpoint information indicating the selected viewpoint selected using the three-dimensional model in the display device 103 is received from the display device 103 (S107 in FIG. 5 and S271 in FIG. 15).
- the free viewpoint image is an image viewed from the selected viewpoint.
- the user of the display device 103 can select the viewpoint of the free viewpoint image using the three-dimensional model. Therefore, the user can easily select the viewpoint of the free viewpoint image.
- the free viewpoint image generation apparatus 102 includes a processor and a memory, and the processor performs the above processing using the memory.
- the display device 103 performs the processing shown in FIG. First, the display device 103 receives the location information indicating the first location where the free viewpoint image is generated and the first time information indicating the first time zone where the free viewpoint image is generated as the free viewpoint image generation device 102. (S311: S101B in FIG. 5, S216 in FIG. 6). Next, the display device 103 receives from the free viewpoint image generation device 102 a first video (camera video) in which the first place is captured in the first time zone (S312: S104 in FIG. 5, S104 in FIG. 10). S241).
- a first video camera video
- the display device 103 displays a first user interface (distributed video display screen 402) for selecting a designated time included in the first time zone using the first video (S313: S104 in FIG. 5). , S243 in FIG. Next, the display device 103 transmits designated time information indicating the designated time selected via the first user interface to the free viewpoint image generating device 102 (S314: S104 in FIG. 5, S245 in FIG. 10).
- the display device 103 receives the free viewpoint image at the designated time of the first place from the free viewpoint image generation device 102 (S315: S108 in FIG. 5, S281 in FIG. 16). Next, the display device 103 displays a free viewpoint image (S316: S108 in FIG. 5 and S283 in FIG. 16).
- the user of the display device 103 can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
- the display device 103 transmits the second time information indicating the second time zone including the first time zone to the free viewpoint image generation device 102 before transmitting the first time information to the free viewpoint image generation device. (S101A in FIG. 5 and S216 in FIG. 6).
- the display device 103 receives thumbnail images at a plurality of times included in the second time zone from the free viewpoint image generation device 102 (S101A in FIG. 5 and S211 in FIG. 6).
- a second user interface (setting screen 401B) for designating the first time zone using the thumbnail image is displayed (S101A in FIG. 5, S212 and S213 in FIG. 6).
- the first time zone is specified via the second user interface.
- the user of the display device 103 can designate a detailed time zone using the thumbnail image. Therefore, the user can easily select a target for generating a free viewpoint image.
- the display device 103 further receives the three-dimensional model at the designated time of the first place from the free viewpoint image generation device 102 (S106 in FIG. 5 and S261 in FIG. 13), and selects the selected viewpoint using the three-dimensional model.
- the third user interface three-dimensional model display screen 403 for selecting (S106 in FIG. 5, S263 in FIG. 13), and viewpoint information indicating the selected viewpoint selected via the third user interface Is transmitted to the free viewpoint image generation apparatus 102 (S106 in FIG. 5 and S265 in FIG. 13).
- the free viewpoint image is an image viewed from the selected viewpoint.
- the user of the display device 103 can select the viewpoint of the free viewpoint image using the three-dimensional model. Therefore, the user can easily select the viewpoint of the free viewpoint image.
- the display device 103 includes a processor and a memory, and the processor performs the above processing using the memory.
- each processing unit included in the free viewpoint image generation system according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.
- circuits are not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor.
- An FPGA Field Programmable Gate Array
- reconfigurable processor that can reconfigure the connection and setting of the circuit cells inside the LSI may be used.
- each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component.
- Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.
- the present disclosure may be realized as various methods such as a free viewpoint image generation method, a free viewpoint image generation method executed by a free viewpoint image generation system, a free viewpoint image generation device, or a display device, or a free viewpoint image display method.
- division of functional blocks in the block diagram is an example, and a plurality of functional blocks can be realized as one functional block, a single functional block can be divided into a plurality of functions, or some functions can be transferred to other functional blocks. May be.
- functions of a plurality of functional blocks having similar functions may be processed in parallel or time-division by a single hardware or software.
- the free viewpoint image generation system according to one or more aspects has been described based on the embodiment.
- the present disclosure is not limited to this embodiment. Unless the scope of the present disclosure deviates from, the scope of one or a plurality of aspects includes those in which various modifications conceived by those skilled in the art have been made in this embodiment, and those constructed by combining components in different embodiments. May be included.
- This disclosure can be applied to a free viewpoint image generation system and the like.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Processing Or Creating Images (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
This free viewpoint image generation method comprises: receiving, from a display device (103), place information indicating a first place that is a generation object of a free viewpoint image, and a first time information indicating a first time zone of the generation object of the free viewpoint image (S301); transmitting, to the display device (103), a first video obtained by photographing the first place in the first time zone (S302); receiving, from the display device (103), designated time information indicating a designated time included in the first time zone and selected using the first video in the display device (103) (S303); generating a free viewpoint image at the designated time in the first place (S304); and transmitting the free viewpoint image to the display device (103) (S305).
Description
本開示は、自由視点画像生成方法、自由視点画像表示方法、自由視点画像生成装置及び表示装置に関する。
The present disclosure relates to a free viewpoint image generation method, a free viewpoint image display method, a free viewpoint image generation device, and a display device.
多視点映像の配信方法として、特許文献1には、複数の視点から撮影された映像を視点移動に連動して配信する技術が開示されている。
As a multi-view video distribution method, Patent Document 1 discloses a technique for distributing video shot from a plurality of viewpoints in conjunction with viewpoint movement.
また、特定シーンを複数の校正済みカメラにより撮影した映像群を用いてそのシーンの三次元モデルを生成し、当該三次元モデルを用いて、そのシーンを自由な視点から見た自由視点画像(自由視点映像)を生成する技術が知られている。
In addition, a 3D model of the scene is generated using a group of images obtained by photographing a specific scene with a plurality of calibrated cameras, and a free viewpoint image (free image) of the scene viewed from a free viewpoint using the 3D model. A technique for generating (viewpoint video) is known.
このような、自由視点画像を配信するシステムにおいて、自由視点画像を生成する対象をユーザが容易に選択できることが望まれている。
In such a system for distributing a free viewpoint image, it is desired that the user can easily select a target for generating the free viewpoint image.
本開示は、自由視点画像を生成する対象をユーザが容易に選択できる自由視点画像生成方法、自由視点画像表示方法、自由視点画像生成装置又は表示装置を提供することを目的とする。
This disclosure is intended to provide a free viewpoint image generation method, a free viewpoint image display method, a free viewpoint image generation apparatus, or a display apparatus that allows a user to easily select a target for generating a free viewpoint image.
本開示の一態様に係る自由視点画像生成方法は、自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報とを表示装置から受信し、前記第1の場所が前記第1の時間帯において撮影された第1映像を前記表示装置に送信し、前記表示装置において前記第1映像を用いて選択された、前記第1時間帯に含まれる指定時刻を示す指定時刻情報を前記表示装置から受信し、前記第1の場所の前記指定時刻における前記自由視点画像を生成し、前記自由視点画像を前記表示装置に送信する。
A free viewpoint image generation method according to an aspect of the present disclosure includes location information indicating a first place where a free viewpoint image is generated and a first time period indicating a first time zone where the free viewpoint image is generated. Information is received from the display device, and the first location is transmitted to the display device in the first time taken in the first time zone, and the display device is selected using the first video. , Receiving specified time information indicating a specified time included in the first time zone from the display device, generating the free viewpoint image at the specified time of the first place, and displaying the free viewpoint image on the display device Send to.
本開示の一態様に係る自由視点画像表示方法は、自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報を自由視点画像生成装置に送信し、前記第1の場所が前記第1の時間帯において撮影された第1映像を前記自由視点画像生成装置から受信し、前記第1映像を用いて前記第1時間帯に含まれる指定時刻を選択するための第1のユーザインタフェースを表示し、前記第1のユーザインタフェースを介して選択された前記指定時刻を示す指定時刻情報を前記自由視点画像生成装置に送信し、前記第1の場所の前記指定時刻における前記自由視点画像を前記自由視点画像生成装置から受信し、前記自由視点画像を表示する。
A free viewpoint image display method according to an aspect of the present disclosure includes: location information indicating a first place where a free viewpoint image is to be generated; and a first time indicating a first time zone where the free viewpoint image is to be generated. Information is transmitted to a free viewpoint image generation device, a first video in which the first location is captured in the first time zone is received from the free viewpoint image generation device, and the first video is used to A first user interface for selecting a designated time included in one hour zone is displayed, and designated time information indicating the designated time selected via the first user interface is displayed in the free viewpoint image generation device. Transmitting, receiving the free viewpoint image at the specified time of the first place from the free viewpoint image generating device, and displaying the free viewpoint image.
なお、これらの全般的または具体的な態様は、システム、方法、集積回路、コンピュータプログラムまたはコンピュータ読み取り可能なCD-ROMなどの記録媒体で実現されてもよく、システム、方法、集積回路、コンピュータプログラム及び記録媒体の任意な組み合わせで実現されてもよい。
These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. The system, method, integrated circuit, computer program Also, any combination of recording media may be realized.
本開示は、自由視点画像を生成する対象をユーザが容易に選択できる自由視点画像生成方法、自由視点画像表示方法、自由視点画像生成装置又は表示装置を提供できる。
This disclosure can provide a free viewpoint image generation method, a free viewpoint image display method, a free viewpoint image generation apparatus, or a display apparatus that allows a user to easily select a target for generating a free viewpoint image.
本開示の一態様に係る自由視点画像生成方法は、自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報とを表示装置から受信し、前記第1の場所が前記第1の時間帯において撮影された第1映像を前記表示装置に送信し、前記表示装置において前記第1映像を用いて選択された、前記第1時間帯に含まれる指定時刻を示す指定時刻情報を前記表示装置から受信し、前記第1の場所の前記指定時刻における前記自由視点画像を生成し、前記自由視点画像を前記表示装置に送信する。
A free viewpoint image generation method according to an aspect of the present disclosure includes location information indicating a first place where a free viewpoint image is generated and a first time period indicating a first time zone where the free viewpoint image is generated. Information is received from the display device, and the first location is transmitted to the display device in the first time taken in the first time zone, and the display device is selected using the first video. , Receiving specified time information indicating a specified time included in the first time zone from the display device, generating the free viewpoint image at the specified time of the first place, and displaying the free viewpoint image on the display device Send to.
これによれば、表示装置のユーザは、第1映像を用いて指定時刻を選択することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
According to this, the user of the display device can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
例えば、前記表示装置から前記第1の時間情報を受信する前に、前記第1の時間帯を含む第2の時間帯を示す第2の時間情報を前記表示装置から受信し、前記第2の時間帯に含まれる複数の時刻におけるサムネイル画像を前記表示装置に送信し、前記第1の時間帯は、前記表示装置において前記サムネイル画像を用いて指定されてもよい。
For example, before receiving the first time information from the display device, the second time information indicating a second time zone including the first time zone is received from the display device, and the second time information is displayed. Thumbnail images at a plurality of times included in a time zone may be transmitted to the display device, and the first time zone may be specified using the thumbnail images on the display device.
これによれば、表示装置のユーザは、サムネイル画像を用いて詳細な時間帯を指定することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
According to this, the user of the display device can designate a detailed time zone using the thumbnail image. Therefore, the user can easily select a target for generating a free viewpoint image.
例えば、前記自由視点画像生成方法は、さらに、前記第1の場所の前記指定時刻における三次元モデルを生成し、前記三次元モデルを前記表示装置に送信し、前記表示装置において前記三次元モデルを用いて選択された選択視点を示す視点情報を前記表示装置から受信し、前記自由視点画像は、前記選択視点から見た画像であってもよい。
For example, the free viewpoint image generation method further generates a three-dimensional model of the first place at the designated time, transmits the three-dimensional model to the display device, and the display device displays the three-dimensional model. The viewpoint information indicating the selected viewpoint selected by use may be received from the display device, and the free viewpoint image may be an image viewed from the selected viewpoint.
これによれば、表示装置のユーザは、三次元モデルを用いて自由視点画像の視点を選択することができる。よって、ユーザは自由視点画像の視点を容易に選択できる。
According to this, the user of the display device can select the viewpoint of the free viewpoint image using the three-dimensional model. Therefore, the user can easily select the viewpoint of the free viewpoint image.
本開示の一態様に係る自由視点画像表示方法は、自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報を自由視点画像生成装置に送信し、前記第1の場所が前記第1の時間帯において撮影された第1映像を前記自由視点画像生成装置から受信し、前記第1映像を用いて前記第1時間帯に含まれる指定時刻を選択するための第1のユーザインタフェースを表示し、前記第1のユーザインタフェースを介して選択された前記指定時刻を示す指定時刻情報を前記自由視点画像生成装置に送信し、前記第1の場所の前記指定時刻における前記自由視点画像を前記自由視点画像生成装置から受信し、前記自由視点画像を表示する。
A free viewpoint image display method according to an aspect of the present disclosure includes: location information indicating a first place where a free viewpoint image is to be generated; and a first time indicating a first time zone where the free viewpoint image is to be generated. Information is transmitted to a free viewpoint image generation device, a first video in which the first location is captured in the first time zone is received from the free viewpoint image generation device, and the first video is used to A first user interface for selecting a designated time included in one hour zone is displayed, and designated time information indicating the designated time selected via the first user interface is displayed in the free viewpoint image generation device. Transmitting, receiving the free viewpoint image at the specified time of the first place from the free viewpoint image generating device, and displaying the free viewpoint image.
これによれば、表示装置のユーザは、第1映像を用いて指定時刻を選択することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
According to this, the user of the display device can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
例えば、前記第1の時間情報を前記自由視点画像生成装置に送信する前に、前記第1の時間帯を含む第2の時間帯を示す第2の時間情報を前記自由視点画像生成装置に送信し、前記第2の時間帯に含まれる複数の時刻におけるサムネイル画像を前記自由視点画像生成装置から受信し、前記サムネイル画像を用いて前記第1の時間帯を指定するための第2のユーザインタフェースを表示し、前記第1の時間帯は、前記第2のユーザインタフェースを介して指定されてもよい。
For example, before transmitting the first time information to the free viewpoint image generation device, second time information indicating a second time zone including the first time zone is transmitted to the free viewpoint image generation device. A second user interface for receiving thumbnail images at a plurality of times included in the second time zone from the free viewpoint image generation device and designating the first time zone using the thumbnail images. And the first time zone may be specified via the second user interface.
これによれば、表示装置のユーザは、サムネイル画像を用いて詳細な時間帯を指定することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
According to this, the user of the display device can designate a detailed time zone using the thumbnail image. Therefore, the user can easily select a target for generating a free viewpoint image.
例えば、前記自由視点画像表示方法は、さらに、前記第1の場所の前記指定時刻における三次元モデルを前記自由視点画像生成装置から受信し、前記三次元モデルを用いて選択視点を選択するための第3のユーザインタフェースを表示し、前記第3のユーザインタフェースを介して選択された前記選択視点を示す視点情報を前記自由視点画像生成装置に送信し、前記自由視点画像は、前記選択視点から見た画像であってもよい。
For example, the free viewpoint image display method further receives a three-dimensional model at the specified time of the first place from the free viewpoint image generation device, and selects a selected viewpoint using the three-dimensional model. A third user interface is displayed, and viewpoint information indicating the selected viewpoint selected via the third user interface is transmitted to the free viewpoint image generation device, and the free viewpoint image is viewed from the selected viewpoint. It may be an image.
これによれば、表示装置のユーザは、三次元モデルを用いて自由視点画像の視点を選択することができる。よって、ユーザは自由視点画像の視点を容易に選択できる。
According to this, the user of the display device can select the viewpoint of the free viewpoint image using the three-dimensional model. Therefore, the user can easily select the viewpoint of the free viewpoint image.
また、本開示の一態様に係る自由視点画像生成装置は、複数の画像から三次元モデルを生成する自由視点画像生成装置であって、プロセッサと、メモリとを備え、前記プロセッサは、前記メモリを用いて、自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報とを表示装置から受信し、前記第1の場所が前記第1の時間帯において撮影された第1映像を前記表示装置に送信し、前記表示装置において前記第1映像を用いて選択された、前記第1時間帯に含まれる指定時刻を示す指定時刻情報を前記表示装置から受信し、前記第1の場所の前記指定時刻における前記自由視点画像を生成し、前記自由視点画像を前記表示装置に送信する。
A free viewpoint image generation device according to an aspect of the present disclosure is a free viewpoint image generation device that generates a three-dimensional model from a plurality of images, and includes a processor and a memory, and the processor includes the memory. And receiving from the display device location information indicating a first location for generating a free viewpoint image and first time information indicating a first time zone for generating the free viewpoint image; A designated time included in the first time zone, which is selected using the first video in the display device by transmitting a first video imaged in one place in the first time zone to the display device. Is received from the display device, the free viewpoint image at the specified time at the first location is generated, and the free viewpoint image is transmitted to the display device.
これによれば、表示装置のユーザは、第1映像を用いて指定時刻を選択することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
According to this, the user of the display device can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
また、本開示の一態様に係る表示装置は、プロセッサと、メモリとを備え、前記プロセッサは、前記メモリを用いて、自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報を自由視点画像生成装置に送信し、前記第1の場所が前記第1の時間帯において撮影された第1映像を前記自由視点画像生成装置から受信し、前記第1映像を用いて前記第1時間帯に含まれる指定時刻を選択するための第1のユーザインタフェースを表示し、前記第1のユーザインタフェースを介して選択された前記指定時刻を示す指定時刻情報を前記自由視点画像生成装置に送信し、前記第1の場所の前記指定時刻における前記自由視点画像を前記自由視点画像生成装置から受信し、前記自由視点画像を表示する。
In addition, a display device according to an aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to store location information indicating a first location where a free viewpoint image is generated, and the free information. First time information indicating a first time zone for which a viewpoint image is to be generated is transmitted to a free viewpoint image generating device, and the first video in which the first place is captured in the first time zone is the free time. A first user interface received from the viewpoint image generation device and used to select a designated time included in the first time zone using the first video is displayed and selected via the first user interface. The designated time information indicating the designated time is transmitted to the free viewpoint image generating device, the free viewpoint image at the designated time of the first place is received from the free viewpoint image generating device, and the free view image is received. To display the image.
これによれば、表示装置のユーザは、第1映像を用いて指定時刻を選択することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
According to this, the user of the display device can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
なお、これらの包括的または具体的な態様は、システム、方法、集積回路、コンピュータプログラムまたはコンピュータ読み取り可能なCD-ROMなどの記録媒体で実現されてもよく、システム、方法、集積回路、コンピュータプログラム及び記録媒体の任意な組み合わせで実現されてもよい。
Note that these comprehensive or specific modes may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, and computer program. Also, any combination of recording media may be realized.
以下、実施の形態について、図面を参照しながら具体的に説明する。なお、以下で説明する実施の形態は、いずれも本開示の一具体例を示すものである。以下の実施の形態で示される数値、形状、材料、構成要素、構成要素の配置位置及び接続形態、ステップ、ステップの順序などは、一例であり、本開示を限定する主旨ではない。また、以下の実施の形態における構成要素のうち、最上位概念を示す独立請求項に記載されていない構成要素については、任意の構成要素として説明される。
Hereinafter, embodiments will be specifically described with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.
本実施の形態では、多視点カメラ及びその映像を利用しての自由視点画像の生成を容易に実現することを可能とする自由視点画像生成システムについて説明する。
In the present embodiment, a free viewpoint image generation system capable of easily realizing generation of a free viewpoint image using a multi-view camera and its video will be described.
複数のカメラ映像及び自由視点画像を利用することで様々な方向からの映像を取得及び生成することが可能となり、多様な視聴者のニーズに応えた映像(例えば、選手Aをアップにした映像、又は俯瞰映像等)を提供できる。
By using multiple camera videos and free viewpoint images, it is possible to acquire and generate videos from various directions, and videos that meet the needs of various viewers (for example, videos with player A up, Or a bird's-eye view video).
図1は、自由視点画像生成システムの概要を示す図である。例えば、校正済みのカメラ(例えば固定カメラ)を用いて同一空間を多視点から撮影することにより撮影する空間を三次元再構成できる(三次元空間再構成)。この三次元再構成されたデータを用いて、トラッキング、シーン解析、及び映像レンダリングを行うことで、任意の視点(自由視点カメラ)から見た映像を生成できる。これにより、次世代広域監視システム、及び自由視点画像生成システムを実現できる。
FIG. 1 is a diagram showing an outline of a free viewpoint image generation system. For example, it is possible to reconstruct a 3D space by photographing the same space from multiple viewpoints using a calibrated camera (for example, a fixed camera) (3D space reconstruction). By using this three-dimensional reconstructed data for tracking, scene analysis, and video rendering, a video viewed from an arbitrary viewpoint (free viewpoint camera) can be generated. Thereby, a next generation wide area monitoring system and a free viewpoint image generation system can be realized.
しかしながら、ユーザの求めるタイミングにおける自由視点画像を生成するためには、生成したい時刻の指定及び視点の設定が必要となる。しかし、時刻と視点という異なる次元の設定を、ユーザが手軽に行うことは容易ではない。
However, in order to generate a free viewpoint image at the timing requested by the user, it is necessary to specify the time to be generated and set the viewpoint. However, it is not easy for the user to easily set different dimensions of time and viewpoint.
これを実現するためには、時刻の設定及び視点の設定を独立した試行かつ連続的に行うことが必要となる。
In order to realize this, it is necessary to perform time setting and viewpoint setting independently and continuously.
本開示の一態様に係る自由視点画像生成システムでは、表示装置は、場所及び時間帯の情報を用いて自由視点画像を生成したい時刻を含むカメラ映像の取得し、カメラ映像から自由視点時刻を生成したい時刻を指定し、三次元モデルを用いて自由視点画像を生成したい視点を指定する。これにより、ユーザの求める自由視点画像の取得を容易に行えるシステムを実現できる。
In the free viewpoint image generation system according to an aspect of the present disclosure, the display device acquires a camera video including a time at which a free viewpoint image is to be generated by using the location and time zone information, and generates a free viewpoint time from the camera video. Specify the time you want to generate, and use the 3D model to specify the viewpoint from which you want to generate a free viewpoint image. Thereby, the system which can acquire the free viewpoint image which a user asks easily can be realized.
これにより、ユーザはカメラ映像から自由視点画像を取得するまでの複雑な処理手順を、一連の処理を行うことで実施できるようになるため、自由視点画像の利用を手軽に行うことが可能となる。
As a result, the user can carry out a complicated processing procedure from acquiring a free viewpoint image from a camera image by performing a series of processes, and thus it is possible to easily use the free viewpoint image. .
以下、本実施の形態に係る自由視点画像生成システム100の構成を説明する。図2は、本実施の形態に係る自由視点画像生成システム100の構成例を示す図である。この自由視点画像生成システム100は、複数のカメラ101と、自由視点画像生成装置102と、複数の表示装置103とを含む。
Hereinafter, the configuration of the free viewpoint image generation system 100 according to the present embodiment will be described. FIG. 2 is a diagram illustrating a configuration example of the free viewpoint image generation system 100 according to the present embodiment. The free viewpoint image generation system 100 includes a plurality of cameras 101, a free viewpoint image generation apparatus 102, and a plurality of display apparatuses 103.
複数のカメラ101は、同一シーンを異なる視点から撮影することで複数のカメラ映像を生成する。例えば、複数のカメラ101は同期撮影を行ってもよい。または、複数のカメラ101は、映像に時刻情報を埋め込む、又は映像にフレーム順序を示すインデックス情報を付与してもよい。
The plurality of cameras 101 generate a plurality of camera images by photographing the same scene from different viewpoints. For example, the plurality of cameras 101 may perform synchronous shooting. Alternatively, the plurality of cameras 101 may embed time information in the video or give index information indicating the frame order to the video.
自由視点画像生成装置102は、自由視点画像を生成するサーバであり、複数のカメラ101で撮影された複数のカメラ映像を取得し、当該複数のカメラ映像を用いて、三次元モデル及び自由視点画像を生成する。また、自由視点画像生成装置102は、ネットワーク104を介して複数の表示装置103に接続されている。自由視点画像生成装置102は、カメラ映像、三次元モデル及び自由視点画像を表示装置103に送信する。具体的には、自由視点画像生成装置102は、表示装置103から送信された時刻及び視点の情報に基づき、それぞれ三次元モデル及び自由視点画像を生成し、生成した三次元モデル及び自由視点画像を表示装置103へ送信する。この際、自由視点画像生成装置102は、三次元モデルを利用して自由視点画像を生成してもよいし、2枚以上の画像から映像を補間することで自由視点映像を生成してもよい。
The free viewpoint image generation device 102 is a server that generates a free viewpoint image, acquires a plurality of camera images captured by a plurality of cameras 101, and uses the plurality of camera images to generate a three-dimensional model and a free viewpoint image. Is generated. The free viewpoint image generation device 102 is connected to a plurality of display devices 103 via the network 104. The free viewpoint image generation device 102 transmits the camera video, the three-dimensional model, and the free viewpoint image to the display device 103. Specifically, the free viewpoint image generation device 102 generates a three-dimensional model and a free viewpoint image based on the time and viewpoint information transmitted from the display device 103, respectively. Transmit to the display device 103. At this time, the free viewpoint image generation device 102 may generate a free viewpoint image using a three-dimensional model, or may generate a free viewpoint image by interpolating images from two or more images. .
表示装置103は、自由視点画像生成装置102から送信されたカメラ映像、三次元モデル、及び自由視点画像を表示することでユーザに提示する端末である。また、表示装置103は、カメラ映像から時刻を選択し、三次元モデルから視点を選択し、それぞれの情報を自由視点画像生成装置102に送信する。
The display device 103 is a terminal that presents to the user by displaying the camera video, the three-dimensional model, and the free viewpoint image transmitted from the free viewpoint image generating device 102. Further, the display device 103 selects time from the camera video, selects a viewpoint from the three-dimensional model, and transmits each information to the free viewpoint image generation device 102.
なお、自由視点画像生成装置102と表示装置103との間ではデータのやり取りができればよく、通信方式は特に限定されない。例えば、遠隔地間では、インターネットを介して通信が行われてもよいし、実験室などにおいてLAN内で通信が行われてもよい。
Note that the communication method is not particularly limited as long as data can be exchanged between the free viewpoint image generation device 102 and the display device 103. For example, communication between remote locations may be performed via the Internet, or may be performed within a LAN in a laboratory or the like.
図3は、本実施の形態に係る自由視点画像生成装置102のブロック図である。自由視点画像生成装置102は、全体設定部201と、カメラ映像配信部202と、三次元モデル生成部203と、自由視点画像生成部204と、通信部205と、設定保存部206と、カメラ映像保存部207と、カメラパラメータ保存部208と、三次元モデル保存部209と、自由視点画像保存部210とを備える。
FIG. 3 is a block diagram of the free viewpoint image generation apparatus 102 according to the present embodiment. The free viewpoint image generation apparatus 102 includes an overall setting unit 201, a camera video distribution unit 202, a 3D model generation unit 203, a free viewpoint image generation unit 204, a communication unit 205, a setting storage unit 206, and a camera video. A storage unit 207, a camera parameter storage unit 208, a 3D model storage unit 209, and a free viewpoint image storage unit 210 are provided.
全体設定部201は、自由視点画像を生成する際に利用する映像の時間及び場所である設定時間及び設定場所を設定し、設定した設定時間及び設定場所を示す情報を設定保存部206に保存する。例えば、事故現場の検証の際には、事故発生の瞬間を映している前後数分の時刻が設定時間として設定され、事故発生場所が設定場所として設定される。例えば、全体設定部201は、通信部205を通して表示装置103から指定された時間及び場所を設定時間及び設定場所として設定する。または、全体設定部201は、群衆を検知するなどの画像処理を用いて事故を認識することで自動的に設定場所及び設定時間を設定し、設定した設定場所及び設定情報を表示装置103に通知してもよい。
The overall setting unit 201 sets a setting time and a setting location, which are the time and place of a video used when generating a free viewpoint image, and stores information indicating the set setting time and the setting location in the setting storage unit 206. . For example, at the time of verification of an accident site, several minutes before and after reflecting the moment of occurrence of the accident are set as the set time, and the accident occurrence place is set as the set place. For example, the overall setting unit 201 sets the time and place designated from the display device 103 through the communication unit 205 as the setting time and setting place. Alternatively, the overall setting unit 201 automatically sets a setting location and a setting time by recognizing an accident using image processing such as detecting a crowd, and notifies the display device 103 of the set setting location and setting information. May be.
カメラ映像配信部202は、全体設定部201で設定された設定時間及び設定場所に対応するカメラ映像を取得し、取得したカメラ映像を表示装置103に送信する。
The camera video distribution unit 202 acquires the camera video corresponding to the set time and the set location set by the overall setting unit 201 and transmits the acquired camera video to the display device 103.
三次元モデル生成部203は、表示装置103から、三次元モデルを生成したい時刻を受け取り、該当時刻の三次元モデルを生成する。また、三次元モデル生成部203は、生成した三次元モデルを表示装置103に送信するとともに、当該三次元モデルを三次元モデル保存部209に保存する。例えば、三次元モデル生成部203は、SfM(Structure from Motion)などの画像処理技術を用いて三次元再構成を行うことで三次元モデルを生成する。または、三次元でもる生成部203は、カメラ映像に深度情報が含まれる場合には、当該深度情報を用いて三次元モデルを生成する。
The 3D model generation unit 203 receives the time at which the 3D model is to be generated from the display device 103, and generates a 3D model at the corresponding time. In addition, the 3D model generation unit 203 transmits the generated 3D model to the display device 103 and stores the 3D model in the 3D model storage unit 209. For example, the 3D model generation unit 203 generates a 3D model by performing 3D reconstruction using an image processing technique such as SfM (Structure from Motion). Alternatively, the three-dimensional generation unit 203 generates a three-dimensional model using the depth information when the camera video includes depth information.
自由視点画像生成部204は、表示装置103から送信された視点情報を受け取り、その視点から見た自由視点画像を生成し、生成した自由視点画像を表示装置103に送信する。また、自由視点画像生成部204は、自由視点画像保存部210に生成した自由視点画像を保存する。例えば、自由視点画像生成部204は、三次元モデルを用いて自由視点画像を生成する。または、自由視点画像生成部204は、モーフィング処理のようにカメラ間の画像を二次元上で補間することで自由視点画像を生成する。なお、自由視点画像(自由視点映像)は、静止画像であってもよし、動画像であってもよい。また、動画像とは、ある視点から見た特定シーンの時系列変化を示す動画像であってもよいし、指定の時刻において、視点位置を連続的に変更した動画像であってもよいし、これらの組み合わせであってもよい。
The free viewpoint image generation unit 204 receives the viewpoint information transmitted from the display device 103, generates a free viewpoint image viewed from the viewpoint, and transmits the generated free viewpoint image to the display device 103. The free viewpoint image generation unit 204 stores the generated free viewpoint image in the free viewpoint image storage unit 210. For example, the free viewpoint image generation unit 204 generates a free viewpoint image using a three-dimensional model. Alternatively, the free viewpoint image generation unit 204 generates a free viewpoint image by interpolating images between cameras two-dimensionally as in morphing processing. Note that the free viewpoint image (free viewpoint video) may be a still image or a moving image. The moving image may be a moving image showing a time-series change of a specific scene viewed from a certain viewpoint, or may be a moving image in which the viewpoint position is continuously changed at a specified time. A combination of these may also be used.
通信部205は、全体設定部201、カメラ映像配信部202、三次元モデル生成部203、及び自由視点画像生成部204と、表示装置103とのデータのやり取りを行う。
The communication unit 205 exchanges data with the display device 103 and the overall setting unit 201, the camera video distribution unit 202, the 3D model generation unit 203, and the free viewpoint image generation unit 204.
設定保存部206は、自由視点画像を生成したい時間帯及び場所に関する全体処理に係る情報を格納する。
The setting storage unit 206 stores information related to the entire process regarding the time zone and place where the free viewpoint image is to be generated.
カメラ映像保存部207は、カメラ101で撮影されたカメラ映像を格納する。具体的には、カメラ映像保存部207は、カメラ101から直接取得したカメラ映像を保存する。または、カメラ映像保存部207は、他の装置を介して間接的に取得したカメラ映像を保存していてもよい。
The camera video storage unit 207 stores the camera video shot by the camera 101. Specifically, the camera video storage unit 207 stores the camera video acquired directly from the camera 101. Alternatively, the camera video storage unit 207 may store a camera video acquired indirectly through another device.
カメラパラメータ保存部208は、カメラ映像保存部207に格納されたカメラ映像を撮影したカメラ101の三次元位置及び姿勢情報を含むカメラパラメータを格納する。この三次元位置及び姿勢情報は、例えば、カメラ101が備えるGPS又はジャイロセンサーなどで取得された情報である。または、自由視点画像生成装置102は、カメラ映像を元にSfMなどの画像処理技術を用いてこの三次元位置及び姿勢情報を推定してもよい。
The camera parameter storage unit 208 stores camera parameters including the three-dimensional position and orientation information of the camera 101 that captured the camera video stored in the camera video storage unit 207. The three-dimensional position and orientation information is information acquired by, for example, a GPS or gyro sensor provided in the camera 101. Alternatively, the free viewpoint image generation apparatus 102 may estimate the three-dimensional position and orientation information using an image processing technique such as SfM based on the camera video.
三次元モデル保存部209は、三次元モデル生成部203で生成された三次元モデルを格納する。
The 3D model storage unit 209 stores the 3D model generated by the 3D model generation unit 203.
自由視点画像保存部210は、自由視点画像生成部204で生成された自由視点画像を格納する。
The free viewpoint image storage unit 210 stores the free viewpoint image generated by the free viewpoint image generation unit 204.
図4は、本実施の形態に係る表示装置103のブロック図である。表示装置103は、設定画面生成部301と、配信映像表示画面生成部302と、三次元モデル表示画面生成部303と、自由視点画像表示画面生成部304と、通信部305と、画面制御部306と、画面表示部307と、カメラ映像保存部308と、三次元モデル保存部309と、自由視点画像保存部310とを備える。
FIG. 4 is a block diagram of the display device 103 according to the present embodiment. The display device 103 includes a setting screen generation unit 301, a distribution video display screen generation unit 302, a 3D model display screen generation unit 303, a free viewpoint image display screen generation unit 304, a communication unit 305, and a screen control unit 306. A screen display unit 307, a camera video storage unit 308, a 3D model storage unit 309, and a free viewpoint image storage unit 310.
設定画面生成部301は、自由視点画像を生成する時刻を含んだ映像を取得するための時間帯情報及び場所情報を含む開始信号を生成するためのインタフェースである設定画面を生成する。なお、画面とは、静止した画像に限らず、動画像を含んでもよいし、時間的に表示形態が変化するアイコン及び操作部等を含んでもよい。
The setting screen generation unit 301 generates a setting screen that is an interface for generating a start signal including time zone information and location information for acquiring a video including a time for generating a free viewpoint image. Note that the screen is not limited to a static image, and may include a moving image, or may include an icon, an operation unit, and the like whose display form changes with time.
配信映像表示画面生成部302は、取得したカメラ映像から自由視点画像を生成する時刻を選択するためのインタフェースである配信映像表示画面を生成する。
The distribution video display screen generation unit 302 generates a distribution video display screen that is an interface for selecting a time for generating a free viewpoint image from the acquired camera video.
三次元モデル表示画面生成部303は、取得した三次元モデルを用いて自由視点画像を生成するための視点を決定するためのインタフェースである三次元モデル表示画面を生成する。
The 3D model display screen generation unit 303 generates a 3D model display screen that is an interface for determining a viewpoint for generating a free viewpoint image using the acquired 3D model.
自由視点画像表示画面生成部304は、取得した自由視点画像を表示する画面である自由視点画像表示画面を生成する。
The free viewpoint image display screen generation unit 304 generates a free viewpoint image display screen that is a screen for displaying the acquired free viewpoint image.
通信部305は、自由視点画像生成装置102と通信し、設定画面生成部301、配信映像表示画面生成部302、三次元モデル表示画面生成部303、及び自由視点画像表示画面生成部304で用いられる情報の送受信を行う。
The communication unit 305 communicates with the free viewpoint image generation device 102 and is used by the setting screen generation unit 301, the distribution video display screen generation unit 302, the 3D model display screen generation unit 303, and the free viewpoint image display screen generation unit 304. Send and receive information.
画面制御部306は、設定画面生成部301、配信映像表示画面生成部302、三次元モデル表示画面生成部303、及び自由視点画像表示画面生成部304にて生成された表示用の画面のいずれを、画面表示部307に表示するか、及びその表示タイミングを制御する。
The screen control unit 306 includes any one of the display screens generated by the setting screen generation unit 301, the distribution video display screen generation unit 302, the 3D model display screen generation unit 303, and the free viewpoint image display screen generation unit 304. Or display on the screen display unit 307 and control the display timing.
また、画面制御部306は、画面表示部307に対して行われたユーザの操作等を取得し、設定画面生成部301、配信映像表示画面生成部302、三次元モデル表示画面生成部303、及び自由視点画像表示画面生成部304へユーザの操作等の情報を送る。
In addition, the screen control unit 306 acquires a user operation performed on the screen display unit 307, and the setting screen generation unit 301, the distribution video display screen generation unit 302, the 3D model display screen generation unit 303, and Information such as user operations is sent to the free viewpoint image display screen generation unit 304.
画面表示部307は、画面制御部306から取得した画面を表示することで各種情報をユーザに提示する。
The screen display unit 307 presents various information to the user by displaying the screen acquired from the screen control unit 306.
カメラ映像保存部308は、自由視点画像生成装置102から送信されたカメラ映像を保存しておくストレージである。
The camera video storage unit 308 is a storage for storing the camera video transmitted from the free viewpoint image generation device 102.
三次元モデル保存部309は、自由視点画像生成装置102から送信された三次元モデルを保存しておくストレージである。
The 3D model storage unit 309 is a storage that stores the 3D model transmitted from the free viewpoint image generation apparatus 102.
自由視点画像保存部310は、自由視点画像生成装置102から送信された自由視点画像を保存しておくストレージである。
The free viewpoint image storage unit 310 is a storage for storing the free viewpoint image transmitted from the free viewpoint image generation apparatus 102.
なお、カメラ映像保存部308、三次元モデル保存部309、及び自由視点画像保存部310は、一時的及び長期間各々のデータを保存できればよく、メモリのような短期的なストレージでもHDD(ハードディスクドライブ)のような長期間データを保持できるものでもよい。
The camera image storage unit 308, the 3D model storage unit 309, and the free viewpoint image storage unit 310 only need to be able to store data temporarily and for a long period of time. ) That can hold data for a long period of time.
次に、本実施の形態に係る自由視点画像生成システム100の動作を説明する。図5は、本実施の形態に係る自由視点画像生成システム100の動作を示すシーケンス図である。
Next, the operation of the free viewpoint image generation system 100 according to the present embodiment will be described. FIG. 5 is a sequence diagram showing the operation of the free viewpoint image generation system 100 according to the present embodiment.
まず、表示装置103は、開始信号を自由視点画像生成装置102に送信する(S101A、S101B)。図6は、この開始信号送信処理(S101)のフローチャートである。
First, the display device 103 transmits a start signal to the free viewpoint image generation device 102 (S101A, S101B). FIG. 6 is a flowchart of the start signal transmission process (S101).
まず、表示装置103は、自由視点画像生成装置102から候補情報を受信したか否かを判定する(S211)。ここで候補情報は、例えば、自由視点画像を生成する対象の場所及び時間(時間帯又は時刻)の複数の候補を示す情報である。
First, the display apparatus 103 determines whether candidate information has been received from the free viewpoint image generation apparatus 102 (S211). Here, the candidate information is, for example, information indicating a plurality of candidates for the location and time (time zone or time) of the target for generating the free viewpoint image.
例えば、図5に示すステップS101Aでは、表示装置103は、候補情報を受信していない(S211でNo)。この場合、表示装置103は、候補情報を含まない設定画面401Aを表示する。図7Aは、この設定画面401Aの一例を示す図である。例えば、設定画面401Aは、時間情報が入力される時間入力部411と、場所情報が入力される場所入力部412とを含む。
For example, in step S101A shown in FIG. 5, the display device 103 has not received candidate information (No in S211). In this case, the display device 103 displays a setting screen 401A that does not include candidate information. FIG. 7A is a diagram showing an example of the setting screen 401A. For example, the setting screen 401A includes a time input unit 411 for inputting time information and a place input unit 412 for inputting place information.
なお、ここで示す設定画面401Aは一例であり、表示装置103は、自由視点画像を生成する時間及び場所を特定するための入力情報を入力可能なインタフェースを有すればよい。例えば、図7Aでは、入力情報の一例として、時刻情報及び場所情報が入力される例を示すが、自由視点画像を生成したい時間を含んだ映像をユーザが選択できればよく、時刻情報及び場所情報以外の情報が利用されてもよい。また、入力方式として任意の方式を用いてよく、例えば、音声入力等が用いられてもよい。
Note that the setting screen 401A shown here is merely an example, and the display device 103 only needs to have an interface through which input information for specifying a time and place for generating a free viewpoint image can be input. For example, FIG. 7A shows an example in which time information and location information are input as an example of input information, but it is sufficient if the user can select a video including a time for which a free viewpoint image is to be generated. May be used. Further, any method may be used as the input method, for example, voice input or the like may be used.
また、図5に示すステップS101Bでは、表示装置103は、候補情報を受信している(S211でYes)。この場合、表示装置103は、受信した候補情報を含む設定画面401Bを生成し(S212)、生成した設定画面401Bを表示する(S213)。図7Bは、この設定画面401Bの一例を示す図である。設定画面401Bは、設定画面401Aの内容に加え、候補情報を表示する候補情報表示部413を含む。候補情報は、例えば、同一地点の映像の一定時刻(例えば5分間隔)ごとのサムネイル画像である。これにより、ユーザは、アクシデント等が起こった時刻を推定することが可能となるので、ユーザによる時間の選択を容易できる。なお、ユーザによりサムネイル画像が選択されることで当該サムネイル画像に対応する時間が自動的に選択されてもよい。
In step S101B shown in FIG. 5, the display apparatus 103 receives candidate information (Yes in S211). In this case, the display device 103 generates a setting screen 401B including the received candidate information (S212), and displays the generated setting screen 401B (S213). FIG. 7B is a diagram showing an example of the setting screen 401B. The setting screen 401B includes a candidate information display unit 413 that displays candidate information in addition to the contents of the setting screen 401A. The candidate information is, for example, a thumbnail image at a certain time (for example, every 5 minutes) of the video at the same point. Thereby, since the user can estimate the time when the accident or the like has occurred, the user can easily select the time. Note that the time corresponding to the thumbnail image may be automatically selected by the user selecting the thumbnail image.
設定画面401A又は401Bが表示された後、表示装置103は、設定画面401A又は401Bを用いてユーザにより入力された時間を示す時間情報を取得する(S214)。例えば、ユーザは、時間入力部411に時間情報を入力する。なお、この時間情報は、秒単位の入力であってもよいし、時間単位の大まかな入力であってもよい。つまり、時間情報は時刻を示してもよいし、時間帯を示してもよい。
After the setting screen 401A or 401B is displayed, the display device 103 acquires time information indicating the time input by the user using the setting screen 401A or 401B (S214). For example, the user inputs time information to the time input unit 411. Note that this time information may be input in units of seconds or rough input in units of time. That is, the time information may indicate a time or a time zone.
また、表示装置103は、設定画面401A又は401Bを用いてユーザにより入力された場所情報を取得する(S215)。例えば、ユーザは、場所入力部412に場所情報を入力する。例えば、場所情報は、住所であってもよいし、交差点又は建造物の固有名等でもあってよいし、緯度経度であってもよいし、利用者が任意で設定したID(識別子)などあってもよい。また、場所情報は、湾岸エリア又は工業地区などの大まかな場所を示してもよい。例えば、図5に示すステップS101Aでは、大まかな時間が指定され、ステップS101Bでは詳細な時間が指定される。
In addition, the display device 103 acquires location information input by the user using the setting screen 401A or 401B (S215). For example, the user inputs location information to the location input unit 412. For example, the location information may be an address, a unique name of an intersection or a building, a latitude / longitude, or an ID (identifier) arbitrarily set by the user. May be. The location information may indicate a rough location such as a bay area or an industrial area. For example, in step S101A shown in FIG. 5, a rough time is specified, and a detailed time is specified in step S101B.
次に、表示装置103は、ステップS214で取得された時間情報と、ステップS215で取得された場所情報とを含む開始信号を自由視点画像生成装置102に送信する(S216)。
Next, the display apparatus 103 transmits a start signal including the time information acquired in step S214 and the location information acquired in step S215 to the free viewpoint image generation apparatus 102 (S216).
図8は、開始信号を受信した自由視点画像生成装置102の全体設定部201の動作(S102)を示すフローチャートである。まず、全体設定部201は、表示装置103から開始信号を受信したか否かを判定する(S221)。全体設定部201は、開始信号を受信しなかった場合(S221でNo)、再度ステップS221を行う。つまり、全体設定部201は、開始信号を受信するまで待ち状態を続ける。
FIG. 8 is a flowchart showing the operation (S102) of the overall setting unit 201 of the free viewpoint image generation apparatus 102 that has received the start signal. First, the overall setting unit 201 determines whether a start signal has been received from the display device 103 (S221). If the start signal is not received (No in S221), the overall setting unit 201 performs Step S221 again. That is, the entire setting unit 201 continues to wait until it receives a start signal.
全体設定部201は、開始信号を受信した場合(S221でYes)、受信した開始信号に含まれる情報(場所情報及び時間情報)が一意であるか否かを判定する(S222)。ここで、情報が一意であるとは、場所情報及び時間情報によって、表示装置103に送信する映像を一意に決定できる状態を示す。具体例としては、この状態とは、13:00:00の交差点Aの映像のように指定されている場合である。
When the start signal is received (Yes in S221), the overall setting unit 201 determines whether information (location information and time information) included in the received start signal is unique (S222). Here, the information is unique indicates a state in which a video to be transmitted to the display device 103 can be uniquely determined based on location information and time information. As a specific example, this state is a case where the image is specified like an image of the intersection A at 13:00.
情報が一意である場合(S222でYes)、全体設定部201は、開始信号に含まれた情報で一意に決定された映像の配信を開始するための配信開始信号をカメラ映像配信部202に送信する(S223)。例えば、図5に示すステップS102Bでは、情報が一意であり、全体設定部201は、配信開始信号をカメラ映像配信部202に送信する。
If the information is unique (Yes in S222), the overall setting unit 201 transmits a distribution start signal for starting distribution of the video uniquely determined by the information included in the start signal to the camera video distribution unit 202. (S223). For example, in step S <b> 102 </ b> B illustrated in FIG. 5, the information is unique, and the overall setting unit 201 transmits a distribution start signal to the camera video distribution unit 202.
一方、情報が一意でない場合(S222でNo)、全体設定部201は、開始信号に含まれる曖昧な情報を明確化するために必要な情報を取得又は生成することで候補情報を生成する(S224)。ここでいう、曖昧な情報とは、時間情報に関しては13:00:00ではなく、昼ごろ又は午前中といった一定時間以上の範囲を含んでいる場合である。また、その際の候補情報の一例としては、昼ごろの11:00から13:00程度の10分刻みでのサムネイル画像群がある。
On the other hand, if the information is not unique (No in S222), the overall setting unit 201 generates candidate information by acquiring or generating information necessary to clarify the ambiguous information included in the start signal (S224). ). The ambiguous information here refers to a case where the time information includes a range of a certain time or more, such as around noon or in the morning, instead of 13:00. In addition, as an example of the candidate information at that time, there is a thumbnail image group in about 10 minutes from 11:00 to 13:00 around noon.
次に、全体設定部201は、生成された候補情報を表示装置103に送信する(S225)。これにより、全体設定部201は、開始信号に含まれる情報を一意にする作業の補助を行うことができる。例えば、図5に示すステップS102Aでは、情報が一意ではなく、全体設定部201は、候補情報を表示装置103に送信する。
Next, the overall setting unit 201 transmits the generated candidate information to the display device 103 (S225). Thereby, the whole setting part 201 can assist the operation | work which makes the information contained in a start signal unique. For example, in step S <b> 102 </ b> A illustrated in FIG. 5, the information is not unique, and the entire setting unit 201 transmits candidate information to the display device 103.
このように、ユーザが大まかな入力を行った場合は絞り込み作業が行われる。例えば、時間情報として大まかな時間が入力された場合、表示装置103は、例えば、5分単位のサムネイル画像を取得し一覧表示する。これにより、ユーザは、絞り込み作業を行うことが可能なユーザインタフェースを通して正確な時間を探すことができるので、正確な時間情報を探索することができる。
In this way, when the user makes a rough input, a narrowing operation is performed. For example, when a rough time is input as time information, the display device 103 acquires, for example, thumbnail images in units of 5 minutes and displays them as a list. As a result, the user can search for accurate time through a user interface capable of performing the narrowing down operation, and thus can search for accurate time information.
なお、ここでは、主に、時間情報が一意でない場合の例を示したが、場所情報に対しても同様の手法を適用できる。例えば、エリアなどの大まかな場所が指定された場合、表示装置103は、時間情報で指定される時間の、当該エリアに含まれる複数の地点のサムネイル画像を表示する。ユーザは、表示されたサムネイル画像から所望の地点を選択することで、詳細な場所情報を選択できる。
Note that, here, an example in which time information is not unique is mainly shown, but the same method can be applied to location information. For example, when a rough place such as an area is designated, the display device 103 displays thumbnail images of a plurality of points included in the area at the time designated by the time information. The user can select detailed location information by selecting a desired location from the displayed thumbnail images.
また、これらの情報は、ユーザが直接入力する必要はなく、例えば、事故情報をまとめたデータベースなどと連携して、そのデータベースに含まれる時間情報及び場所情報を利用してもよい。
In addition, it is not necessary for the user to directly input these information. For example, time information and location information included in the database may be used in cooperation with a database that summarizes accident information.
図9は、配信開始信号を受信したカメラ映像配信部202の動作(S103)を示すフローチャートである。まず、カメラ映像配信部202は、配信開始信号を受信したか否かを判定する(S231)。配信開始信号を受信していない場合(S231でNo)、カメラ映像配信部202は、ステップS231を再度行う。つまり、カメラ映像配信部202は、配信開始信号を受信するまで待ち状態を続ける。
FIG. 9 is a flowchart showing the operation (S103) of the camera video distribution unit 202 that has received the distribution start signal. First, the camera video distribution unit 202 determines whether a distribution start signal has been received (S231). When the distribution start signal has not been received (No in S231), the camera video distribution unit 202 performs Step S231 again. That is, the camera video distribution unit 202 continues to wait until a distribution start signal is received.
カメラ映像配信部202は、配信開始信号を受信した場合(S231でYes)、配信開始信号に含まれる時間情報及び場所情報を取得し、配信する映像に関する場所及び時間を決定する(S232)。例えば、カメラ映像配信部202は、場所情報及び時間情報で示される場所及び時間を、配信する映像の場所及び時間に決定する。
When receiving the distribution start signal (Yes in S231), the camera video distribution unit 202 acquires the time information and the location information included in the distribution start signal, and determines the location and time related to the video to be distributed (S232). For example, the camera video distribution unit 202 determines the location and time indicated by the location information and time information as the location and time of the video to be distributed.
次に、カメラ映像配信部202は、ステップS232において決定した場所及び時間に該当する映像をカメラ映像保存部207から取得する(S233)。なお、現時刻が指定された場合のように、カメラ101から直接カメラ映像を取得できる場合などには、カメラ映像配信部202は、カメラ101からカメラ映像を直接取得してもよい。最後に、カメラ映像配信部202は、取得したカメラ映像を表示装置103に送信する(S234)。
Next, the camera video distribution unit 202 acquires the video corresponding to the location and time determined in step S232 from the camera video storage unit 207 (S233). Note that the camera video distribution unit 202 may directly acquire the camera video from the camera 101 when the camera video can be acquired directly from the camera 101, such as when the current time is designated. Finally, the camera video distribution unit 202 transmits the acquired camera video to the display device 103 (S234).
図10は、カメラ映像を受信した表示装置103における処理(S104)を示すフローチャートである。まず、表示装置103は、自由視点画像生成装置102からカメラ映像を受信したか否かを判定する(S241)。表示装置103は、カメラ映像を受信していない場合(S241でNo)、再度ステップS241を行う。つまり、表示装置103は、カメラ映像を受信するまで待ち状態を続ける。
FIG. 10 is a flowchart showing the process (S104) in the display device 103 that has received the camera video. First, the display device 103 determines whether a camera video has been received from the free viewpoint image generation device 102 (S241). If the display device 103 has not received the camera video (No in S241), the display device 103 performs Step S241 again. That is, the display device 103 continues to wait until a camera video is received.
表示装置103は、カメラ映像を受信した場合(S241でYes)、受信したカメラ映像を利用して配信映像表示画面402を生成する(S242)。次に、表示装置103は、生成した配信映像表示画面402を画面制御部306及び画面表示部307を用いて、ユーザに提示する(S243)。
When the display device 103 receives the camera video (Yes in S241), the display device 103 generates the distribution video display screen 402 using the received camera video (S242). Next, the display device 103 presents the generated distribution video display screen 402 to the user using the screen control unit 306 and the screen display unit 307 (S243).
図11は、配信映像表示画面402の一例を示す図である。配信映像表示画面402は、受信したカメラ映像が表示される映像表示部421と、三次元モデルを生成したい時刻を指定するための操作部422とを含む。
FIG. 11 is a diagram illustrating an example of the distribution video display screen 402. The distribution video display screen 402 includes a video display unit 421 on which the received camera video is displayed, and an operation unit 422 for designating a time at which a 3D model is to be generated.
なお、表示装置103は、複数のカメラ映像を受信し、映像表示部421に複数のカメラ映像が表示されてもよい。この複数のカメラ映像は、例えば、同一時間帯において同一の場面を異なる視点から撮影したカメラ映像である。なお、表示装置103は、映像表示部421に、複数のカメラ映像の全てを同時に表示してもよいし、複数のカメラ映像のうち、1視点以上の映像を選択して表示してもよい。また、映像を選択した場合には、ユーザが映像を切り替えることができるようなインタフェースが設けられてもよい。
Note that the display device 103 may receive a plurality of camera videos and display a plurality of camera videos on the video display unit 421. The plurality of camera videos are, for example, camera videos obtained by shooting the same scene from different viewpoints in the same time zone. Note that the display device 103 may display all of the plurality of camera images on the image display unit 421 at the same time, or may select and display one or more viewpoint images from among the plurality of camera images. In addition, when a video is selected, an interface may be provided so that the user can switch the video.
操作部422は、映像の再生及び停止を行う機能、並びに三次元モデルを生成する時刻である指定時刻を決定する機能を有すればよい。ユーザによる指定時刻の決定を容易にするために、操作部422は、スライドバー又はタイムコードを用いた映像中の時刻表示の可視化を行う機能を有してもよい。また、操作部422は、早送り又は巻き戻しといった再生速度又は再生方向の変更を行う機能を有してもよい。
The operation unit 422 may have a function of playing and stopping video and a function of determining a designated time that is a time for generating a three-dimensional model. In order to facilitate the determination of the designated time by the user, the operation unit 422 may have a function of visualizing the time display in the video using a slide bar or a time code. Further, the operation unit 422 may have a function of changing the playback speed or playback direction such as fast forward or rewind.
例えば、ユーザは、操作部422のスライドバーを指定時刻に合わせ、決定操作(例えば決定ボタンの操作)を行うことで指定時刻を入力する。なお、指定時刻の入力方法はこれに限定されず、任意の方法でよい。
For example, the user inputs the designated time by adjusting the slide bar of the operation unit 422 to the designated time and performing a decision operation (for example, a decision button operation). The method for inputting the designated time is not limited to this, and any method may be used.
次に、表示装置103は、ユーザにより配信映像表示画面402を介して入力された指定時刻を取得する(S244)。次に、表示装置103は、取得した指定時刻を示す指定時刻情報を自由視点画像生成装置102に送信する(S245)。
Next, the display device 103 acquires the designated time input by the user via the distribution video display screen 402 (S244). Next, the display device 103 transmits specified time information indicating the acquired specified time to the free viewpoint image generation device 102 (S245).
図12は、指定時刻情報を受信した自由視点画像生成装置102の三次元モデル生成部203による処理(S105)のフローチャートである。まず、三次元モデル生成部203は、表示装置103から指定時刻情報を受信したか否かを判定する(S251)。三次元モデル生成部203は、指定時刻情報を受信していない場合(S251でNo)、再度ステップS251を行う。つまり、三次元モデル生成部203は、指定時刻情報を受信するまで待ち状態を続ける。
FIG. 12 is a flowchart of the process (S105) by the three-dimensional model generation unit 203 of the free viewpoint image generation apparatus 102 that has received the specified time information. First, the three-dimensional model generation unit 203 determines whether designated time information has been received from the display device 103 (S251). If the specified time information has not been received (No in S251), the three-dimensional model generation unit 203 performs Step S251 again. That is, the three-dimensional model generation unit 203 continues to wait until receiving specified time information.
三次元モデル生成部203は、指定時刻情報を受信した場合(S251でYes)、指定時刻情報で示される指定時刻の三次元モデルを生成する(S252)。例えば、三次元モデル生成部203は、PMVS(Patch-based Multi View Stereo)を用いて三次元モデルを生成する。具体的には、三次元モデル生成部203は、物体のコーナーなどの画像内の特徴のある特徴点を元に、画像間で同一特徴点を決定し、決定した特徴点に対して、カメラの三次元姿勢を含むカメラパラメータを用いた三次元測量を行うことで、三次元モデルを生成する。または、三次元モデル生成部203は、視体積交差法を用いて三次元モデルを生成してもよい。つまり、三次元モデル生成部203は、各カメラで写っているオブジェクト領域を三次元上で統合することによって三次元モデルを生成してもよい。
3D model generation unit 203 receives the specified time information (Yes in S251), and generates a 3D model at the specified time indicated by the specified time information (S252). For example, the 3D model generation unit 203 generates a 3D model using PMVS (Patch-based Multi View Stereo). Specifically, the three-dimensional model generation unit 203 determines the same feature point between images based on feature points in the image such as a corner of an object, and the camera feature is determined for the determined feature points. A 3D model is generated by performing a 3D survey using camera parameters including a 3D posture. Alternatively, the three-dimensional model generation unit 203 may generate a three-dimensional model using a visual volume intersection method. That is, the three-dimensional model generation unit 203 may generate a three-dimensional model by integrating the object regions captured by each camera in three dimensions.
最後に、三次元モデル生成部203は、生成した三次元モデルを表示装置103に送信する(S253)。
Finally, the 3D model generation unit 203 transmits the generated 3D model to the display device 103 (S253).
なお、表示装置103に三次元モデルに関する情報を送信する手法は如何なる手法でもよい。例えば、三次元モデル生成部203は、一度、テキスト等の何らかのファイルに三次元モデルを書き出し、そのファイルを送信してもよい。または、三次元モデル生成部203は、生成した三次元モデルのデータを適宜ストリーミング送信してもよい。前者では、自由視点画像生成装置102内及び表示装置103内での時刻ごとの三次元モデルの管理等が容易になる。後者の方法では、表示装置103への送信に必要となる時間を短縮することが可能である。
Note that any method may be used for transmitting information related to the three-dimensional model to the display device 103. For example, the three-dimensional model generation unit 203 may once write the three-dimensional model in some file such as text and transmit the file. Alternatively, the 3D model generation unit 203 may appropriately stream the generated 3D model data. In the former, it becomes easy to manage a three-dimensional model for each time in the free viewpoint image generation apparatus 102 and the display apparatus 103. In the latter method, the time required for transmission to the display device 103 can be shortened.
なお、三次元モデル生成部203は、指定時刻の三次元モデルを生成したのちに、その指定時刻の前後の時刻の三次元モデルを生成し始めてもよい。例えば、指定時刻が時刻nである場合、三次元モデル生成部203は、時刻nの三次元モデルを生成及び送信したのちに、自動的に時刻n+1及び時刻n-1の三次元モデルを生成してもよい。また、三次元モデル生成部203は、生成した時刻n+1及び時刻n-1の三次元モデルを表示装置103に送信してもよいし、送信しなくてもよい。これによると、指定時刻の前後数フレーム、具体例としては交通事故の瞬間の前後等の三次元モデルが事前に作成される。ユーザは、指定時刻の周辺時刻の自由視点画像も見る可能性が高い。よって、このように前もって準備を行うことで、三次元モデルを生成するための待ち時間が不要となり、ユーザの待ち時間を減らすことが可能となる。
The 3D model generation unit 203 may start generating a 3D model at a time before and after the specified time after generating a 3D model at the specified time. For example, when the designated time is time n, the 3D model generation unit 203 automatically generates a 3D model at time n + 1 and time n−1 after generating and transmitting the 3D model at time n. May be. Further, the three-dimensional model generation unit 203 may or may not transmit the generated three-dimensional model at time n + 1 and time n−1 to the display device 103. According to this, a three-dimensional model is created in advance such as several frames before and after the designated time, specifically, before and after the moment of a traffic accident. The user has a high possibility of seeing a free viewpoint image at a time around the designated time. Therefore, by making preparations in advance in this way, the waiting time for generating the three-dimensional model is unnecessary, and the waiting time of the user can be reduced.
なお、三次元モデル生成部203は、ステップS252において、指定時刻とその前後の時刻の三次元モデルを生成し、ステップS253において、生成した複数時刻の三次元モデルを表示装置103に送信してもよい。または、指定時刻情報において複数時刻又は時間帯が指定され、三次元モデル生成部203は、指定された複数時刻又は時間帯の三次元モデルを生成し、生成した三次元モデルを表示装置103に送信してもよい。
Note that the three-dimensional model generation unit 203 generates a three-dimensional model at the designated time and the times before and after it in step S252, and transmits the generated three-dimensional model at a plurality of times to the display device 103 in step S253. Good. Alternatively, multiple times or time zones are specified in the specified time information, and the 3D model generation unit 203 generates a 3D model of the specified multiple times or time zones and transmits the generated 3D model to the display device 103. May be.
図13は、三次元モデルを受信した表示装置103における処理(S106)を示すフローチャートである。まず、表示装置103は、自由視点画像生成装置102から三次元モデルを受信したか否かを判定する(S261)。表示装置103は、三次元モデルを受信していない場合(S261でNo)、再度ステップS261を行う。つまり、表示装置103は、三次元モデルを受信するまで待ち状態を続ける。
FIG. 13 is a flowchart showing the process (S106) in the display device 103 that has received the three-dimensional model. First, the display apparatus 103 determines whether a three-dimensional model has been received from the free viewpoint image generation apparatus 102 (S261). When the display device 103 has not received the three-dimensional model (No in S261), the display device 103 performs Step S261 again. That is, the display device 103 continues to wait until a three-dimensional model is received.
表示装置103は、三次元モデルを受信した場合(S261でYes)受信した三次元モデルを利用して三次元モデル表示画面403を生成する(S262)。次に、表示装置103は、生成した三次元モデル表示画面403を、画面制御部306及び画面表示部307を用いて、ユーザに提示する(S263)。
When receiving the 3D model (Yes in S261), the display device 103 generates the 3D model display screen 403 using the received 3D model (S262). Next, the display device 103 presents the generated 3D model display screen 403 to the user using the screen control unit 306 and the screen display unit 307 (S263).
図14は、三次元モデル表示画面403の一例を示す図である。三次元モデル表示画面403は、受信した三次元モデルを表示するための三次元モデル表示部431と、三次元モデル表示部431に表示される三次元モデルを操作するための表示操作部432と、現時点において表示している三次元モルの視点を、自由視点画像生成装置102に送る視点情報として決定するため視点決定部433とを含む。
FIG. 14 is a diagram illustrating an example of the three-dimensional model display screen 403. The 3D model display screen 403 includes a 3D model display unit 431 for displaying the received 3D model, a display operation unit 432 for operating the 3D model displayed on the 3D model display unit 431, A viewpoint determination unit 433 is included to determine the viewpoint of the three-dimensional mole currently displayed as viewpoint information to be sent to the free viewpoint image generation apparatus 102.
三次元モデル表示部431は、三次元モデルを所定の視点から見た二次元画像を表示する。例えば、表示装置103は、表示する初期視点を、配信映像表示画面402に表示されていたカメラ映像と同じ視点に設定する。または、表示装置103は、三次元モデル中の座標情報などを解析し、注目オブジェクト(例えば、最も大きいオブジェクト)などがよく見えるように初期視点を設定してもよい。例えば、表示装置103は、注目オブジェクトが画像の真ん中に表示されるように初期視点を設置してもよい。
The 3D model display unit 431 displays a 2D image obtained by viewing the 3D model from a predetermined viewpoint. For example, the display device 103 sets the initial viewpoint to be displayed to the same viewpoint as the camera video displayed on the distribution video display screen 402. Alternatively, the display device 103 may analyze the coordinate information in the three-dimensional model and set the initial viewpoint so that the object of interest (for example, the largest object) can be seen well. For example, the display device 103 may set the initial viewpoint so that the object of interest is displayed in the middle of the image.
表示操作部432は、三次元モデル表示部431に表示される画像の視点(仮想カメラ位置)を変更する操作をユーザが行うための機能を有する。具体的には、視点及び注視点の移動及び回転などの制御を行うことにより、三次元モデルの見える位置及び大きさ等をユーザの期待通りに変更できればよい。すなわち、インタフェースに関しては特に限定されず、キーボードによって視点の移動を制御してもよいし、図14に示すようにボタン等のインタフェースを用意してもよい。または、三次元モデル表示部431上をタッチ操作することで視点が制御されてもよい。
The display operation unit 432 has a function for the user to perform an operation of changing the viewpoint (virtual camera position) of the image displayed on the 3D model display unit 431. Specifically, it is only necessary to change the position and size of the three-dimensional model as expected by the user by controlling the movement and rotation of the viewpoint and the gazing point. That is, the interface is not particularly limited, and the movement of the viewpoint may be controlled by a keyboard, or an interface such as a button may be prepared as shown in FIG. Alternatively, the viewpoint may be controlled by touching the 3D model display unit 431.
視点決定部433は、現在の三次元モデル表示部431に表示されている三次元モデルの見え方状況を再現できる視点を自由視点画像生成装置102に送信する視点として決定するための操作部である。
The viewpoint determination unit 433 is an operation unit for determining a viewpoint capable of reproducing the appearance state of the 3D model currently displayed on the 3D model display unit 431 as a viewpoint to be transmitted to the free viewpoint image generation apparatus 102. .
なお、視点決定部433は、三次元モデル表示部431又は表示操作部432に含まれてもよい。つまり、ユーザが三次元モデル表示部431又は表示操作部432に対して予め定められた操作を行うことで、視点が決定されてもよい。
Note that the viewpoint determination unit 433 may be included in the three-dimensional model display unit 431 or the display operation unit 432. That is, the viewpoint may be determined by the user performing a predetermined operation on the 3D model display unit 431 or the display operation unit 432.
上記のように、表示装置103は、三次元モデル表示画面403に対するユーザの操作により指定された視点を示す視点情報を取得する(S264)。最後に、表示装置103は、取得した視点情報を自由視点画像生成装置102に送信する(S265)。
As described above, the display device 103 acquires viewpoint information indicating a viewpoint designated by a user operation on the three-dimensional model display screen 403 (S264). Finally, the display apparatus 103 transmits the acquired viewpoint information to the free viewpoint image generation apparatus 102 (S265).
図15は、視点情報を受信した自由視点画像生成装置102の自由視点画像生成部204の処理(S107)を示すフローチャートである。まず、自由視点画像生成部204は、自由視点画像を生成するための視点情報を表示装置103から受信したか否かを判定する(S271)。自由視点画像生成部204は、視点情報を受信していない場合(S271でNo)、再度ステップS271を行う。つまり、自由視点画像生成部204は、視点情報を受信するまで待ち状態を続ける。
FIG. 15 is a flowchart showing the processing (S107) of the free viewpoint image generation unit 204 of the free viewpoint image generation apparatus 102 that has received the viewpoint information. First, the free viewpoint image generation unit 204 determines whether or not viewpoint information for generating a free viewpoint image has been received from the display device 103 (S271). When the viewpoint information has not been received (No in S271), the free viewpoint image generation unit 204 performs Step S271 again. That is, the free viewpoint image generation unit 204 continues to wait until it receives viewpoint information.
自由視点画像生成部204は、視点情報を受信した場合(S271でYes)、受信した視点情報で示される視点から見た、指定時刻における自由視点画像を生成する(S272)。例えば、自由視点画像生成部204は、三次元モデルを利用して実際に三次元空間を想定し、指定された視点からの見えをシミュレーションしてレンダリングすることで自由視点画像を生成する。または、自由視点画像生成部204は、指定された視点に近接している2又は3個のカメラ映像を補間することで自由視点画像を生成してもよい。
When the viewpoint information is received (Yes in S271), the free viewpoint image generation unit 204 generates a free viewpoint image at a specified time viewed from the viewpoint indicated by the received viewpoint information (S272). For example, the free viewpoint image generation unit 204 assumes a three-dimensional space using a three-dimensional model, generates a free viewpoint image by simulating and rendering the appearance from a specified viewpoint. Alternatively, the free viewpoint image generation unit 204 may generate a free viewpoint image by interpolating two or three camera images close to the designated viewpoint.
最後に、自由視点画像生成部204は、生成した自由視点画像を表示装置103に送信する(S273)。
Finally, the free viewpoint image generation unit 204 transmits the generated free viewpoint image to the display device 103 (S273).
なお、自由視点画像生成部204は、視点情報に基づいた自由視点画像を生成したのちに、その周辺の視点から見た場合の自由視点画像を生成してもよい。具体的には、自由視点画像生成部204は、三次元モデルを解析して、指定された視点よりもオブジェクトに正対した状態の視点の自由視点画像を生成してもよい。
Note that the free viewpoint image generation unit 204 may generate a free viewpoint image when viewed from the surrounding viewpoints after generating a free viewpoint image based on the viewpoint information. Specifically, the free viewpoint image generation unit 204 may analyze the three-dimensional model and generate a free viewpoint image of a viewpoint that faces the object rather than the designated viewpoint.
また、自由視点画像生成部204は、指定時刻情報で指定された指定時刻の前後の時刻における、視点情報で示される視点から見た自由視点画像を生成してもよい。これによると、ユーザは、指定された視点における時間変化を映像として確認することができるようになり、ユーザへの効果的な情報提示が可能となる。
Also, the free viewpoint image generation unit 204 may generate a free viewpoint image viewed from the viewpoint indicated by the viewpoint information at times before and after the specified time specified by the specified time information. According to this, the user can confirm the time change at the designated viewpoint as an image, and can effectively present information to the user.
また、自由視点画像生成部204は、指定された視点よりも少し広角な画角の自由視点画像を生成してもよい。この場合、表示装置103において表示範囲を、指定された視点と合わせることにより、指定された視点を再現することが可能となる。これによると、表示装置103において、指定した視点が想定より狭かった場合等に、再度の自由視点画像生成処理を行わずにユーザは周辺領域を視認できるようになる。
Also, the free viewpoint image generation unit 204 may generate a free viewpoint image having a field angle slightly wider than the designated viewpoint. In this case, the designated viewpoint can be reproduced by matching the display range with the designated viewpoint in the display device 103. According to this, in the display device 103, when the designated viewpoint is narrower than expected, the user can visually recognize the peripheral area without performing the free viewpoint image generation process again.
また、自由視点画像生成部204は、自由視点画像を生成する際に、指定された視点の自由視点画像の生成のほかに、同一位置かつ同一視線方向でより望遠の画像を生成してもよい。これによると、自由視点画像生成部204は、指定された視点と共に望遠画像を表示装置103に送信することで、表示装置103においてズーム操作を行った際に、これらの画像を切り替えながら表示することができる。これにより、ズーム後も高精細な画像をユーザに提示することが可能となる。
Further, when generating the free viewpoint image, the free viewpoint image generation unit 204 may generate a more telephoto image at the same position and in the same line-of-sight direction in addition to generating the free viewpoint image of the designated viewpoint. . According to this, the free viewpoint image generation unit 204 transmits the telephoto image together with the designated viewpoint to the display device 103 so that when the zoom operation is performed on the display device 103, these images are displayed while being switched. Can do. This makes it possible to present a high-definition image to the user even after zooming.
図16は、自由視点画像を受信した表示装置103の処理(S108)を示すフローチャートである。まず、表示装置103は、自由視点画像生成装置102から自由視点画像を受信したか否かを判定する(S281)。表示装置103は、自由視点画像を受信していない場合(S281でNo)、再度ステップS281を行う。つまり、表示装置103は、自由視点画像を受信するまで待ち状態を続ける。
FIG. 16 is a flowchart showing the processing (S108) of the display device 103 that has received the free viewpoint image. First, the display apparatus 103 determines whether a free viewpoint image has been received from the free viewpoint image generation apparatus 102 (S281). When the display device 103 has not received the free viewpoint image (No in S281), the display device 103 performs Step S281 again. That is, the display device 103 continues to wait until a free viewpoint image is received.
表示装置103は、自由視点画像を受信した場合(S281でYes)、受信した自由視点画像を利用して自由視点画像表示画面404を生成する(S282)。次に、表示装置103は、生成した自由視点画像表示画面404を画面制御部306及び画面表示部307を用いて、ユーザに提示する(S283)。
When the display device 103 receives the free viewpoint image (Yes in S281), the display device 103 generates the free viewpoint image display screen 404 using the received free viewpoint image (S282). Next, the display apparatus 103 presents the generated free viewpoint image display screen 404 to the user using the screen control unit 306 and the screen display unit 307 (S283).
図17は、自由視点画像表示画面404の一例を示す図である。自由視点画像表示画面404は、少なくとも取得した自由視点画像を表示する表示部441を含む。また、自由視点画像表示画面404は、以前に生成した自由視点画像のサムネイル画像等が表示される履歴部442を含んでもよい。
FIG. 17 is a diagram illustrating an example of the free viewpoint image display screen 404. The free viewpoint image display screen 404 includes a display unit 441 that displays at least the acquired free viewpoint image. In addition, the free viewpoint image display screen 404 may include a history unit 442 that displays thumbnail images and the like of previously generated free viewpoint images.
表示部441には、自由視点画像生成装置102から受信した自由視点画像が表示される。表示内容に関しては、静止画像以外にも、同じ視点において連続した時刻で生成された複数の自由視点画像がある場合は、この複数の自由視点画像が自由視点映像として表示されてもよい。また、表示部441において、表示の拡大縮小及び回転などの操作が行われてもよい。この操作は、タッチ操作等のインタフェースで行われてもよいし、キーボード及びマウスを用いた操作で行われてもよい。または、自由視点画像表示画面404は、ボタン等のインタフェースを含み、当該インタフェースを用いて操作が行われてもよい。
The display unit 441 displays the free viewpoint image received from the free viewpoint image generation device 102. Regarding display contents, in addition to still images, when there are a plurality of free viewpoint images generated at consecutive times at the same viewpoint, the plurality of free viewpoint images may be displayed as free viewpoint images. In the display unit 441, operations such as display enlargement / reduction and rotation may be performed. This operation may be performed by an interface such as a touch operation, or may be performed by an operation using a keyboard and a mouse. Alternatively, the free viewpoint image display screen 404 includes an interface such as a button, and an operation may be performed using the interface.
履歴部442は、以前に生成した自由視点画像を後で見返す際などに利用される。例えば、表示装置103は、自由視点画像を取得する度に、取得した自由視点画像を自由視点画像保存部310に保存する。これにより、表示装置103は、履歴内容の表示を行う際に表示装置103内から過去の自由視点画像を取得できるので、再度の自由視点画像の生成処理が不要となる。よって、処理時間の削減を実現できる。
The history unit 442 is used when a free viewpoint image generated previously is viewed later. For example, the display device 103 stores the acquired free viewpoint image in the free viewpoint image storage unit 310 every time a free viewpoint image is acquired. As a result, the display device 103 can acquire a past free viewpoint image from the display device 103 when displaying the history content, so that the process of generating the free viewpoint image again becomes unnecessary. Therefore, reduction in processing time can be realized.
以上のように、本実施の形態に係る自由視点画像生成装置102は、図18に示す処理を行う。まず、自由視点画像生成装置102は、自由視点画像の生成対象の第1の場所を示す場所情報と、自由視点画像の生成対象の第1の時間帯を示す第1の時間情報とを表示装置103から受信する(S301:図5のS102B、図8のS221)。次に、自由視点画像生成装置102は、第1の場所が第1の時間帯において撮影された第1映像(カメラ映像)を表示装置103に送信する(S302:図5のS103、図9のS234)。次に、自由視点画像生成装置102は、表示装置103において第1映像を用いて選択された、第1時間帯に含まれる指定時刻を示す指定時刻情報を表示装置103から受信する(S303:図5のS105、図12のS251)。次に、自由視点画像生成装置102は、第1の場所の指定時刻における自由視点画像を生成し(S304:図5のS107、図15のS272)、自由視点画像を表示装置に送信する(S305:図5のS107、図15のS273)。
As described above, the free viewpoint image generation apparatus 102 according to the present embodiment performs the processing shown in FIG. First, the free viewpoint image generation device 102 displays location information indicating a first location where a free viewpoint image is to be generated and first time information indicating a first time zone where the free viewpoint image is to be generated. 103 (S301: S102B in FIG. 5, S221 in FIG. 8). Next, the free viewpoint image generation apparatus 102 transmits the first video (camera video) taken at the first place in the first time zone to the display device 103 (S302: S103 in FIG. 5, S103 in FIG. 9). S234). Next, the free viewpoint image generation device 102 receives from the display device 103 specified time information indicating a specified time included in the first time zone, which is selected using the first video on the display device 103 (S303: FIG. 5 S105, FIG. 12 S251). Next, the free viewpoint image generation device 102 generates a free viewpoint image at the designated time of the first place (S304: S107 in FIG. 5 and S272 in FIG. 15), and transmits the free viewpoint image to the display device (S305). : S107 in FIG. 5 and S273 in FIG. 15).
これにより、表示装置103のユーザは、第1映像を用いて指定時刻を選択することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
Thereby, the user of the display device 103 can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
また、自由視点画像生成装置102は、表示装置103から第1の時間情報を受信する前に、第1の時間帯を含む第2の時間帯を示す第2の時間情報を表示装置103から受信する(図5のS102A、図8のS221)。次に、自由視点画像生成装置102は、第2の時間帯に含まれる複数の時刻におけるサムネイル画像(候補情報)を表示装置103に送信する(図5のS102A、図8のS225)。第1の時間帯は、表示装置103においてサムネイル画像を用いて指定される。
In addition, the free viewpoint image generation device 102 receives the second time information indicating the second time zone including the first time zone from the display device 103 before receiving the first time information from the display device 103. (S102A in FIG. 5 and S221 in FIG. 8). Next, the free viewpoint image generation device 102 transmits thumbnail images (candidate information) at a plurality of times included in the second time zone to the display device 103 (S102A in FIG. 5 and S225 in FIG. 8). The first time zone is designated using a thumbnail image on the display device 103.
これにより、表示装置103のユーザは、サムネイル画像を用いて詳細な時間帯を指定することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
Thereby, the user of the display device 103 can designate a detailed time zone using the thumbnail image. Therefore, the user can easily select a target for generating a free viewpoint image.
また、自由視点画像生成装置102は、さらに、第1の場所の指定時刻における三次元モデルを生成し(図5のS105、図12のS252)、三次元モデルを表示装置103に送信し(図5のS105、図12のS253)、表示装置103において三次元モデルを用いて選択された選択視点を示す視点情報を表示装置103から受信する(図5のS107、図15のS271)。また、自由視点画像は、選択視点から見た画像である。
The free viewpoint image generation apparatus 102 further generates a three-dimensional model at the designated time of the first place (S105 in FIG. 5 and S252 in FIG. 12), and transmits the three-dimensional model to the display apparatus 103 (FIG. 5 (S105, S253 in FIG. 12), the viewpoint information indicating the selected viewpoint selected using the three-dimensional model in the display device 103 is received from the display device 103 (S107 in FIG. 5 and S271 in FIG. 15). The free viewpoint image is an image viewed from the selected viewpoint.
これにより、表示装置103のユーザは、三次元モデルを用いて自由視点画像の視点を選択することができる。よって、ユーザは自由視点画像の視点を容易に選択できる。
Thereby, the user of the display device 103 can select the viewpoint of the free viewpoint image using the three-dimensional model. Therefore, the user can easily select the viewpoint of the free viewpoint image.
例えば、自由視点画像生成装置102は、プロセッサと、メモリとを備え、プロセッサは、メモリを用いて、上記の処理を行う。
For example, the free viewpoint image generation apparatus 102 includes a processor and a memory, and the processor performs the above processing using the memory.
また、本実施の形態に係る表示装置103は、図19に示す処理を行う。まず、表示装置103は、自由視点画像の生成対象の第1の場所を示す場所情報と、自由視点画像の生成対象の第1の時間帯を示す第1の時間情報を自由視点画像生成装置102に送信する(S311:図5のS101B、図6のS216)。次に、表示装置103は、第1の場所が第1の時間帯において撮影された第1映像(カメラ映像)を自由視点画像生成装置102から受信する(S312:図5のS104、図10のS241)。
Further, the display device 103 according to the present embodiment performs the processing shown in FIG. First, the display device 103 receives the location information indicating the first location where the free viewpoint image is generated and the first time information indicating the first time zone where the free viewpoint image is generated as the free viewpoint image generation device 102. (S311: S101B in FIG. 5, S216 in FIG. 6). Next, the display device 103 receives from the free viewpoint image generation device 102 a first video (camera video) in which the first place is captured in the first time zone (S312: S104 in FIG. 5, S104 in FIG. 10). S241).
次に、表示装置103は、第1映像を用いて第1時間帯に含まれる指定時刻を選択するための第1のユーザインタフェース(配信映像表示画面402)を表示する(S313:図5のS104、図10のS243)。次に、表示装置103は、第1のユーザインタフェースを介して選択された指定時刻を示す指定時刻情報を自由視点画像生成装置102に送信する(S314:図5のS104、図10のS245)。
Next, the display device 103 displays a first user interface (distributed video display screen 402) for selecting a designated time included in the first time zone using the first video (S313: S104 in FIG. 5). , S243 in FIG. Next, the display device 103 transmits designated time information indicating the designated time selected via the first user interface to the free viewpoint image generating device 102 (S314: S104 in FIG. 5, S245 in FIG. 10).
次に、表示装置103は、第1の場所の指定時刻における自由視点画像を自由視点画像生成装置102から受信する(S315:図5のS108、図16のS281)。次に、表示装置103は、自由視点画像を表示する(S316:図5のS108、図16のS283)。
Next, the display device 103 receives the free viewpoint image at the designated time of the first place from the free viewpoint image generation device 102 (S315: S108 in FIG. 5, S281 in FIG. 16). Next, the display device 103 displays a free viewpoint image (S316: S108 in FIG. 5 and S283 in FIG. 16).
これにより、表示装置103のユーザは、第1映像を用いて指定時刻を選択することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
Thereby, the user of the display device 103 can select the designated time using the first video. Therefore, the user can easily select a target for generating a free viewpoint image.
また、表示装置103は、第1の時間情報を自由視点画像生成装置に送信する前に、第1の時間帯を含む第2の時間帯を示す第2の時間情報を自由視点画像生成装置102に送信する(図5のS101A、図6のS216)。表示装置103は、第2の時間帯に含まれる複数の時刻におけるサムネイル画像を自由視点画像生成装置102から受信する(図5のS101A、図6のS211)。サムネイル画像を用いて第1の時間帯を指定するための第2のユーザインタフェース(設定画面401B)を表示する(図5のS101A、図6のS212及びS213)。第1の時間帯は、第2のユーザインタフェースを介して指定される。
In addition, the display device 103 transmits the second time information indicating the second time zone including the first time zone to the free viewpoint image generation device 102 before transmitting the first time information to the free viewpoint image generation device. (S101A in FIG. 5 and S216 in FIG. 6). The display device 103 receives thumbnail images at a plurality of times included in the second time zone from the free viewpoint image generation device 102 (S101A in FIG. 5 and S211 in FIG. 6). A second user interface (setting screen 401B) for designating the first time zone using the thumbnail image is displayed (S101A in FIG. 5, S212 and S213 in FIG. 6). The first time zone is specified via the second user interface.
これにより、表示装置103のユーザは、サムネイル画像を用いて詳細な時間帯を指定することができる。よって、ユーザは自由視点画像を生成する対象を容易に選択できる。
Thereby, the user of the display device 103 can designate a detailed time zone using the thumbnail image. Therefore, the user can easily select a target for generating a free viewpoint image.
また、表示装置103は、さらに、第1の場所の指定時刻における三次元モデルを自由視点画像生成装置102から受信し(図5のS106、図13のS261)、三次元モデルを用いて選択視点を選択するための第3のユーザインタフェース(三次元モデル表示画面403)を表示し(図5のS106、図13のS263)、第3のユーザインタフェースを介して選択された選択視点を示す視点情報を自由視点画像生成装置102に送信する(図5のS106、図13のS265)。また、自由視点画像は、選択視点から見た画像である。
Further, the display device 103 further receives the three-dimensional model at the designated time of the first place from the free viewpoint image generation device 102 (S106 in FIG. 5 and S261 in FIG. 13), and selects the selected viewpoint using the three-dimensional model. The third user interface (three-dimensional model display screen 403) for selecting (S106 in FIG. 5, S263 in FIG. 13), and viewpoint information indicating the selected viewpoint selected via the third user interface Is transmitted to the free viewpoint image generation apparatus 102 (S106 in FIG. 5 and S265 in FIG. 13). The free viewpoint image is an image viewed from the selected viewpoint.
これにより、表示装置103のユーザは、三次元モデルを用いて自由視点画像の視点を選択することができる。よって、ユーザは自由視点画像の視点を容易に選択できる。
Thereby, the user of the display device 103 can select the viewpoint of the free viewpoint image using the three-dimensional model. Therefore, the user can easily select the viewpoint of the free viewpoint image.
例えば、表示装置103は、プロセッサと、メモリとを備え、プロセッサは、メモリを用いて、上記の処理を行う。
For example, the display device 103 includes a processor and a memory, and the processor performs the above processing using the memory.
以上、本開示の実施の形態に係る自由視点画像生成システムについて説明したが、本開示は、この実施の形態に限定されるものではない。
The free viewpoint image generation system according to the embodiment of the present disclosure has been described above, but the present disclosure is not limited to this embodiment.
また、上記実施の形態に係る自由視点画像生成システムに含まれる各処理部は典型的には集積回路であるLSIとして実現される。これらは個別に1チップ化されてもよいし、一部又は全てを含むように1チップ化されてもよい。
Further, each processing unit included in the free viewpoint image generation system according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.
また、集積回路化はLSIに限るものではなく、専用回路又は汎用プロセッサで実現してもよい。LSI製造後にプログラムすることが可能なFPGA(Field Programmable Gate Array)、又はLSI内部の回路セルの接続や設定を再構成可能なリコンフィギュラブル・プロセッサを利用してもよい。
Further, the integration of circuits is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of the circuit cells inside the LSI may be used.
また、上記各実施の形態において、各構成要素は、専用のハードウェアで構成されるか、各構成要素に適したソフトウェアプログラムを実行することによって実現されてもよい。各構成要素は、CPUまたはプロセッサなどのプログラム実行部が、ハードディスクまたは半導体メモリなどの記録媒体に記録されたソフトウェアプログラムを読み出して実行することによって実現されてもよい。
Further, in each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.
また、本開示は、自由視点画像生成システム、自由視点画像生成装置、又は表示装置により実行される自由視点画像生成方法、又は自由視点画像表示方法等の各種方法として実現されてもよい。
Also, the present disclosure may be realized as various methods such as a free viewpoint image generation method, a free viewpoint image generation method executed by a free viewpoint image generation system, a free viewpoint image generation device, or a display device, or a free viewpoint image display method.
また、ブロック図における機能ブロックの分割は一例であり、複数の機能ブロックを一つの機能ブロックとして実現したり、一つの機能ブロックを複数に分割したり、一部の機能を他の機能ブロックに移してもよい。また、類似する機能を有する複数の機能ブロックの機能を単一のハードウェア又はソフトウェアが並列又は時分割に処理してもよい。
In addition, division of functional blocks in the block diagram is an example, and a plurality of functional blocks can be realized as one functional block, a single functional block can be divided into a plurality of functions, or some functions can be transferred to other functional blocks. May be. In addition, functions of a plurality of functional blocks having similar functions may be processed in parallel or time-division by a single hardware or software.
また、フローチャートにおける各ステップが実行される順序は、本開示を具体的に説明するために例示するためのものであり、上記以外の順序であってもよい。また、上記ステップの一部が、他のステップと同時(並列)に実行されてもよい。
In addition, the order in which the steps in the flowchart are executed is for illustration in order to specifically describe the present disclosure, and may be in an order other than the above. Also, some of the above steps may be executed simultaneously (in parallel) with other steps.
以上、一つまたは複数の態様に係る自由視点画像生成システムについて、実施の形態に基づいて説明したが、本開示は、この実施の形態に限定されるものではない。本開示の趣旨を逸脱しない限り、当業者が思いつく各種変形を本実施の形態に施したものや、異なる実施の形態における構成要素を組み合わせて構築される形態も、一つまたは複数の態様の範囲内に含まれてもよい。
As described above, the free viewpoint image generation system according to one or more aspects has been described based on the embodiment. However, the present disclosure is not limited to this embodiment. Unless the scope of the present disclosure deviates from, the scope of one or a plurality of aspects includes those in which various modifications conceived by those skilled in the art have been made in this embodiment, and those constructed by combining components in different embodiments. May be included.
本開示は、自由視点画像生成システム等に適用できる。
This disclosure can be applied to a free viewpoint image generation system and the like.
100 自由視点画像生成システム
101 カメラ
102 自由視点画像生成装置
103 表示装置
104 ネットワーク
201 全体設定部
202 カメラ映像配信部
203 三次元モデル生成部
204 自由視点画像生成部
205、305 通信部
206 設定保存部
207、308 カメラ映像保存部
208 カメラパラメータ保存部
209、309 三次元モデル保存部
210、310 自由視点画像保存部
301 設定画面生成部
302 配信映像表示画面生成部
303 三次元モデル表示画面生成部
304 自由視点画像表示画面生成部
306 画面制御部
307 画面表示部
401A、401B 設定画面
402 配信映像表示画面
403 三次元モデル表示画面
404 自由視点画像表示画面
411 時間入力部
412 場所入力部
413 候補情報表示部
421 映像表示部
422 操作部
431 三次元モデル表示部
432 表示操作部
433 視点決定部
441 表示部
442 履歴部 DESCRIPTION OFSYMBOLS 100 Free viewpoint image generation system 101 Camera 102 Free viewpoint image generation apparatus 103 Display apparatus 104 Network 201 Whole setting part 202 Camera image | video delivery part 203 3D model generation part 204 Free viewpoint image generation part 205,305 Communication part 206 Setting preservation | save part 207 308 Camera image storage unit 208 Camera parameter storage unit 209 309 3D model storage unit 210 310 Free viewpoint image storage unit 301 Setting screen generation unit 302 Distribution video display screen generation unit 303 3D model display screen generation unit 304 Free viewpoint Image display screen generation unit 306 Screen control unit 307 Screen display unit 401A, 401B Setting screen 402 Distribution video display screen 403 Three-dimensional model display screen 404 Free viewpoint image display screen 411 Time input unit 412 Location input unit 413 candidates Information display unit 421 Video display unit 422 Operation unit 431 Three-dimensional model display unit 432 Display operation unit 433 Viewpoint determination unit 441 Display unit 442 History unit
101 カメラ
102 自由視点画像生成装置
103 表示装置
104 ネットワーク
201 全体設定部
202 カメラ映像配信部
203 三次元モデル生成部
204 自由視点画像生成部
205、305 通信部
206 設定保存部
207、308 カメラ映像保存部
208 カメラパラメータ保存部
209、309 三次元モデル保存部
210、310 自由視点画像保存部
301 設定画面生成部
302 配信映像表示画面生成部
303 三次元モデル表示画面生成部
304 自由視点画像表示画面生成部
306 画面制御部
307 画面表示部
401A、401B 設定画面
402 配信映像表示画面
403 三次元モデル表示画面
404 自由視点画像表示画面
411 時間入力部
412 場所入力部
413 候補情報表示部
421 映像表示部
422 操作部
431 三次元モデル表示部
432 表示操作部
433 視点決定部
441 表示部
442 履歴部 DESCRIPTION OF
Claims (8)
- 自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報とを表示装置から受信し、
前記第1の場所が前記第1の時間帯において撮影された第1映像を前記表示装置に送信し、
前記表示装置において前記第1映像を用いて選択された、前記第1時間帯に含まれる指定時刻を示す指定時刻情報を前記表示装置から受信し、
前記第1の場所の前記指定時刻における前記自由視点画像を生成し、
前記自由視点画像を前記表示装置に送信する
自由視点画像生成方法。 Receiving, from a display device, location information indicating a first location for generating a free viewpoint image and first time information indicating a first time zone for generating the free viewpoint image;
Transmitting a first video of the first location taken in the first time zone to the display device;
Received from the display device specified time information indicating a specified time included in the first time zone, selected using the first video in the display device;
Generating the free viewpoint image at the specified time of the first place;
A method for generating a free viewpoint image, wherein the free viewpoint image is transmitted to the display device. - 前記表示装置から前記第1の時間情報を受信する前に、前記第1の時間帯を含む第2の時間帯を示す第2の時間情報を前記表示装置から受信し、
前記第2の時間帯に含まれる複数の時刻におけるサムネイル画像を前記表示装置に送信し、
前記第1の時間帯は、前記表示装置において前記サムネイル画像を用いて指定される
請求項1記載の自由視点画像生成方法。 Before receiving the first time information from the display device, the second time information indicating a second time zone including the first time zone is received from the display device;
Transmitting thumbnail images at a plurality of times included in the second time zone to the display device;
The free viewpoint image generation method according to claim 1, wherein the first time zone is designated by using the thumbnail image on the display device. - 前記自由視点画像生成方法は、さらに、
前記第1の場所の前記指定時刻における三次元モデルを生成し、
前記三次元モデルを前記表示装置に送信し、
前記表示装置において前記三次元モデルを用いて選択された選択視点を示す視点情報を前記表示装置から受信し、
前記自由視点画像は、前記選択視点から見た画像である
請求項1又は2記載の自由視点画像生成方法。 The free viewpoint image generation method further includes:
Generating a three-dimensional model of the first location at the specified time;
Transmitting the three-dimensional model to the display device;
Receiving viewpoint information indicating the selected viewpoint selected using the three-dimensional model in the display device from the display device;
The free viewpoint image generation method according to claim 1, wherein the free viewpoint image is an image viewed from the selected viewpoint. - 自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報を自由視点画像生成装置に送信し、
前記第1の場所が前記第1の時間帯において撮影された第1映像を前記自由視点画像生成装置から受信し、
前記第1映像を用いて前記第1時間帯に含まれる指定時刻を選択するための第1のユーザインタフェースを表示し、
前記第1のユーザインタフェースを介して選択された前記指定時刻を示す指定時刻情報を前記自由視点画像生成装置に送信し、
前記第1の場所の前記指定時刻における前記自由視点画像を前記自由視点画像生成装置から受信し、
前記自由視点画像を表示する
自由視点画像表示方法。 Transmitting location information indicating a first location of a free viewpoint image generation target and first time information indicating a first time zone of the generation target of the free viewpoint image to the free viewpoint image generation device;
Receiving a first video of the first place taken in the first time period from the free viewpoint image generation device;
Displaying a first user interface for selecting a designated time included in the first time zone using the first video;
Transmitting designated time information indicating the designated time selected via the first user interface to the free viewpoint image generating device;
Receiving the free viewpoint image at the specified time of the first place from the free viewpoint image generating device;
A free viewpoint image display method for displaying the free viewpoint image. - 前記第1の時間情報を前記自由視点画像生成装置に送信する前に、前記第1の時間帯を含む第2の時間帯を示す第2の時間情報を前記自由視点画像生成装置に送信し、
前記第2の時間帯に含まれる複数の時刻におけるサムネイル画像を前記自由視点画像生成装置から受信し、
前記サムネイル画像を用いて前記第1の時間帯を指定するための第2のユーザインタフェースを表示し、
前記第1の時間帯は、前記第2のユーザインタフェースを介して指定される
請求項4記載の自由視点画像表示方法。 Before transmitting the first time information to the free viewpoint image generation device, transmit second time information indicating a second time zone including the first time zone to the free viewpoint image generation device,
Receiving thumbnail images at a plurality of times included in the second time zone from the free viewpoint image generation device;
Displaying a second user interface for designating the first time zone using the thumbnail image;
The free viewpoint image display method according to claim 4, wherein the first time zone is designated via the second user interface. - 前記自由視点画像表示方法は、さらに、
前記第1の場所の前記指定時刻における三次元モデルを前記自由視点画像生成装置から受信し、
前記三次元モデルを用いて選択視点を選択するための第3のユーザインタフェースを表示し、
前記第3のユーザインタフェースを介して選択された前記選択視点を示す視点情報を前記自由視点画像生成装置に送信し、
前記自由視点画像は、前記選択視点から見た画像である
請求項4又は5記載の自由視点画像表示方法。 The free viewpoint image display method further includes:
Receiving a three-dimensional model of the first location at the specified time from the free viewpoint image generating device;
Displaying a third user interface for selecting a selected viewpoint using the three-dimensional model;
Transmitting viewpoint information indicating the selected viewpoint selected via the third user interface to the free viewpoint image generation device;
The free viewpoint image display method according to claim 4, wherein the free viewpoint image is an image viewed from the selected viewpoint. - 複数の画像から三次元モデルを生成する自由視点画像生成装置であって、
プロセッサと、
メモリとを備え、
前記プロセッサは、前記メモリを用いて、
自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報とを表示装置から受信し、
前記第1の場所が前記第1の時間帯において撮影された第1映像を前記表示装置に送信し、
前記表示装置において前記第1映像を用いて選択された、前記第1時間帯に含まれる指定時刻を示す指定時刻情報を前記表示装置から受信し、
前記第1の場所の前記指定時刻における前記自由視点画像を生成し、
前記自由視点画像を前記表示装置に送信する
自由視点画像生成装置。 A free viewpoint image generation device that generates a three-dimensional model from a plurality of images,
A processor;
With memory,
The processor uses the memory,
Receiving, from a display device, location information indicating a first location for generating a free viewpoint image and first time information indicating a first time zone for generating the free viewpoint image;
Transmitting a first video of the first location taken in the first time zone to the display device;
Received from the display device specified time information indicating a specified time included in the first time zone, selected using the first video in the display device;
Generating the free viewpoint image at the specified time of the first place;
A free viewpoint image generation device that transmits the free viewpoint image to the display device. - プロセッサと、
メモリとを備え、
前記プロセッサは、前記メモリを用いて、
自由視点画像の生成対象の第1の場所を示す場所情報と、前記自由視点画像の生成対象の第1の時間帯を示す第1の時間情報を自由視点画像生成装置に送信し、
前記第1の場所が前記第1の時間帯において撮影された第1映像を前記自由視点画像生成装置から受信し、
前記第1映像を用いて前記第1時間帯に含まれる指定時刻を選択するための第1のユーザインタフェースを表示し、
前記第1のユーザインタフェースを介して選択された前記指定時刻を示す指定時刻情報を前記自由視点画像生成装置に送信し、
前記第1の場所の前記指定時刻における前記自由視点画像を前記自由視点画像生成装置から受信し、
前記自由視点画像を表示する
表示装置。 A processor;
With memory,
The processor uses the memory,
Transmitting location information indicating a first location of a free viewpoint image generation target and first time information indicating a first time zone of the generation target of the free viewpoint image to the free viewpoint image generation device;
Receiving a first video of the first place taken in the first time period from the free viewpoint image generation device;
Displaying a first user interface for selecting a designated time included in the first time zone using the first video;
Transmitting designated time information indicating the designated time selected via the first user interface to the free viewpoint image generating device;
Receiving the free viewpoint image at the specified time of the first place from the free viewpoint image generating device;
A display device for displaying the free viewpoint image.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2020515567A JP7167134B2 (en) | 2018-04-27 | 2019-04-25 | Free-viewpoint image generation method, free-viewpoint image display method, free-viewpoint image generation device, and display device |
US17/077,423 US20210044793A1 (en) | 2018-04-27 | 2020-10-22 | Generation method for generating free viewpoint image, display method for displaying free viewpoint image, free viewpoint image generation device, and display device |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201862663443P | 2018-04-27 | 2018-04-27 | |
US62/663,443 | 2018-04-27 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/077,423 Continuation US20210044793A1 (en) | 2018-04-27 | 2020-10-22 | Generation method for generating free viewpoint image, display method for displaying free viewpoint image, free viewpoint image generation device, and display device |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019208702A1 true WO2019208702A1 (en) | 2019-10-31 |
Family
ID=68294103
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2019/017658 WO2019208702A1 (en) | 2018-04-27 | 2019-04-25 | Free viewpoint image generation method, free viewpoint image display method, free viewpoint image generation device, and display device |
Country Status (3)
Country | Link |
---|---|
US (1) | US20210044793A1 (en) |
JP (1) | JP7167134B2 (en) |
WO (1) | WO2019208702A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022209362A1 (en) * | 2021-03-31 | 2022-10-06 | 富士フイルム株式会社 | Image processing device, image processing method, and program |
WO2023085075A1 (en) * | 2021-11-12 | 2023-05-19 | ソニーグループ株式会社 | Information processing device and method |
JP2023131043A (en) * | 2022-03-08 | 2023-09-21 | 東京海上日動火災保険株式会社 | Information processor, method for processing information, and program |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2020086983A (en) * | 2018-11-27 | 2020-06-04 | キヤノン株式会社 | Image processing device, image processing method, and program |
JP2022028091A (en) * | 2018-12-19 | 2022-02-15 | ソニーグループ株式会社 | Image processing device, image processing method, and program |
JP2020187706A (en) * | 2019-05-17 | 2020-11-19 | キヤノン株式会社 | Image processing device, image processing system, image processing method, and program |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000224542A (en) * | 1999-01-29 | 2000-08-11 | Hitachi Ltd | Image storage device, monitor system and storage medium |
JP2017228146A (en) * | 2016-06-23 | 2017-12-28 | Kddi株式会社 | Image processing device, image processing method, and computer program |
-
2019
- 2019-04-25 WO PCT/JP2019/017658 patent/WO2019208702A1/en active Application Filing
- 2019-04-25 JP JP2020515567A patent/JP7167134B2/en active Active
-
2020
- 2020-10-22 US US17/077,423 patent/US20210044793A1/en not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000224542A (en) * | 1999-01-29 | 2000-08-11 | Hitachi Ltd | Image storage device, monitor system and storage medium |
JP2017228146A (en) * | 2016-06-23 | 2017-12-28 | Kddi株式会社 | Image processing device, image processing method, and computer program |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022209362A1 (en) * | 2021-03-31 | 2022-10-06 | 富士フイルム株式会社 | Image processing device, image processing method, and program |
WO2023085075A1 (en) * | 2021-11-12 | 2023-05-19 | ソニーグループ株式会社 | Information processing device and method |
JP2023131043A (en) * | 2022-03-08 | 2023-09-21 | 東京海上日動火災保険株式会社 | Information processor, method for processing information, and program |
Also Published As
Publication number | Publication date |
---|---|
JPWO2019208702A1 (en) | 2021-04-30 |
JP7167134B2 (en) | 2022-11-08 |
US20210044793A1 (en) | 2021-02-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7167134B2 (en) | Free-viewpoint image generation method, free-viewpoint image display method, free-viewpoint image generation device, and display device | |
US20210272372A1 (en) | Augmented and virtual reality | |
AU2021203688B2 (en) | Volumetric depth video recording and playback | |
US9851877B2 (en) | Image processing apparatus, image processing method, and computer program product | |
CN106598229B (en) | Virtual reality scene generation method and device and virtual reality system | |
US10917622B2 (en) | Information processing apparatus, display control method, and storage medium | |
JP7017175B2 (en) | Information processing equipment, information processing method, program | |
AU2009257959B2 (en) | 3D content aggregation built into devices | |
US20140181630A1 (en) | Method and apparatus for adding annotations to an image | |
US20210019953A1 (en) | Real-time feedback for surface reconstruction as a service | |
US20230005220A1 (en) | Shooting method, shooting instruction method, shooting device, and shooting instruction device | |
EP2936442A1 (en) | Method and apparatus for adding annotations to a plenoptic light field | |
EP2159756B1 (en) | Point-cloud clip filter | |
CN112740261A (en) | Panoramic light field capture, processing and display | |
JP7497683B2 (en) | VIDEO REPRODUCTION DEVICE, REPRODUCTION METHOD, PROGRAM, AND METHOD FOR GENERATING 3D SHAPE VIDEO DATA | |
EP2093999A1 (en) | Integration of video information | |
WO2020017600A1 (en) | Display control device, display control method and program | |
KR101246844B1 (en) | System for 3D stereo control system and providing method thereof | |
JP2005174024A (en) | Image forming apparatus and method | |
JP2017183816A (en) | Moving image distribution server, moving image output device, moving image distribution system, and moving image distribution method | |
JP2001042259A (en) | Medical image display device | |
KR101563593B1 (en) | Apparatus for automatic cyclic monitoring | |
CN117478931A (en) | Information display method, information display device, electronic equipment and storage medium | |
JP2004234457A (en) | Information processing apparatus, method, program, and storage medium | |
JP2022182836A (en) | Video processing device and its control method, and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19793218 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2020515567 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19793218 Country of ref document: EP Kind code of ref document: A1 |