WO2021199735A1 - 情報処理装置、画像処理システム、情報処理方法 - Google Patents

情報処理装置、画像処理システム、情報処理方法 Download PDF

Info

Publication number
WO2021199735A1
WO2021199735A1 PCT/JP2021/005519 JP2021005519W WO2021199735A1 WO 2021199735 A1 WO2021199735 A1 WO 2021199735A1 JP 2021005519 W JP2021005519 W JP 2021005519W WO 2021199735 A1 WO2021199735 A1 WO 2021199735A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
information processing
section
clip
free
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/005519
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
翔 小倉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to EP21781577.8A priority Critical patent/EP4113453A4/en
Priority to JP2022511634A priority patent/JP7635778B2/ja
Priority to US17/907,854 priority patent/US12219122B2/en
Priority to CN202180023809.4A priority patent/CN115315730A/zh
Publication of WO2021199735A1 publication Critical patent/WO2021199735A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/349—Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking
    • H04N13/351—Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for displaying simultaneously
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8549—Creating video summaries, e.g. movie trailer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • This technology relates to an information processing device, an image processing system, and an information processing method, and particularly to a technology related to processing a free-viewpoint image capable of observing an imaged subject from an arbitrary viewpoint in a three-dimensional space.
  • a free viewpoint image also called a free viewpoint image, a virtual viewpoint image (video), etc.
  • the technology to generate is known.
  • Patent Document 1 discloses a technique relating to the generation of camera work, which can be said to be a movement locus of a viewpoint.
  • the free viewpoint image is also useful as broadcast content, and is also used as a replay image of sports broadcasts, for example.
  • a clip of a few seconds such as a shooting scene is created from an image recorded in real time and broadcast as a replay image.
  • the "clip" refers to an image of a certain scene created by cutting out or further processing the recorded image.
  • this technology proposes a technology that enables faster image production such as clips including free-viewpoint images.
  • the information processing apparatus includes a section specifying processing unit that performs a process of specifying a generation target image section to be generated as a free viewpoint image for a plurality of captured images simultaneously captured by a plurality of imaging devices, and the plurality of image processing units.
  • the target image transmission control unit that controls the transmission of the image data of the generation target image section in each of the captured images as image data used for generating the free viewpoint image in another information processing device, and the received free viewpoint image. It includes an output image generation unit that generates an output image including the image.
  • This information processing device is an information processing device that functions as a main controller for creating an image such as a clip in an image processing system, for example.
  • This information processing device specifies a generation target image section of one frame or a plurality of frames for generating a free-viewpoint image, and outputs image data of only the generation target image section in a plurality of captured images to another information processing device (actually). Send to the free-viewpoint image server) that creates the free-viewpoint image. Then, the created free viewpoint image is received and an output image is generated.
  • the section specifying processing unit performs a process of notifying the other information processing device of the information of the specified generation target image section, and the target image transmission control unit performs the processing. It is conceivable to control the transmission of image data in the generation target image section in response to a request from another information processing device. By notifying another information processing device of the image section to be generated, the other information processing device can specify the image data necessary for generating the free viewpoint image.
  • the section specifying processing unit specifies a section of one frame of the captured image as the generation target image section. This is a process for generating a free viewpoint image in the generation target image section of one frame, that is, a free viewpoint image in which the viewpoint is changed in the state of a still image in which the time is stopped.
  • the section specifying processing unit specifies a section of a plurality of frames of the captured image as the generation target image section. This is a process for generating a free viewpoint image in a generation target image section of a plurality of frames, that is, a free viewpoint image in which the viewpoint is changed in the state of a moving image without stopping the time.
  • the output image generation unit generates an output image in which a front image or a rear image is combined with a free viewpoint image on the time axis. That is, when the information processing device receives the free viewpoint image created by another information processing device, the information processing device connects the preceding and following images to form an output image.
  • the output image generation unit generates an output image as a virtual clip in which a free viewpoint image and a front image or a rear image are virtually combined on a time axis based on playback list information.
  • a free-viewpoint image created by another information processing device is received, the previous and next images are connected to form an output image, but the combination of the images is virtually a connection on the playlist information indicating the playback order. Assumed to be performed.
  • the output image generation unit acquires information for designating a front image or a rear image to be connected to the free viewpoint image from the other information processing device, and based on the information. It is conceivable to generate an output image in which the front image or the rear image is combined with the free viewpoint image on the time axis. That is, the information processing device receives the free viewpoint image created by the other information processing device, and also receives information from the other information processing device that specifies the images before and after the connection with the free viewpoint image.
  • the output image generation unit acquires information for designating the time length of the front image or the rear image connected to the free viewpoint image from the other information processing device, and the information is obtained. It is conceivable to prepare a front image or a rear image based on the above, and generate an output image in which the front image or the rear image is combined with the free viewpoint image on the time axis. That is, the information processing device receives the free viewpoint image created by the other information processing device, and also receives the time length information of the images before and after connecting with the free viewpoint image from the other information processing device.
  • Another information processing device is a target image for acquiring image data of a generation target image section for which a free viewpoint image is generated in each of a plurality of captured images simultaneously captured by a plurality of imaging devices.
  • This information processing device is, for example, an information processing device that functions as a free-viewpoint image server in an image processing system.
  • This information processing device is an information processing device that acquires a plurality of captured images as a section of a scene for generating a free viewpoint image, generates a free viewpoint image based on the captured images, and functions as a main controller for image creation. Send.
  • the target image acquisition unit responds to the notification of the generation target image section and receives a plurality of image data of the generation target image section in each of the plurality of captured images. It is conceivable to make a transmission request. That is, by notifying the image section to be generated, the image section to be acquired is recognized and the transmission of image data of a plurality of viewpoints in the section is requested.
  • the target image acquisition unit acquires one frame of image data as the generation target image section in each of the plurality of captured images. This is a process for generating a free viewpoint image in the generation target image section of one frame, that is, a free viewpoint image in which the viewpoint is changed in the state of a still image in which the time is stopped.
  • the target image acquisition unit acquires image data of a plurality of frames of each of the plurality of captured images as the generation target image section. This is a process for generating a free viewpoint image in a generation target image section of a plurality of frames, that is, a free viewpoint image in which the viewpoint is changed in the state of a moving image without stopping the time.
  • the image generation processing unit generates a free viewpoint image using the viewpoint movement locus information selected from the viewpoint movement locus information stored in advance. ..
  • a plurality of pieces of information indicating the locus of viewpoint movement are prepared in advance so that they can be selected by, for example, an operator's operation.
  • the transmission control unit may control transmission of information for designating a front image or a rear image to be connected to the generated free-viewpoint image to the other information processing apparatus.
  • the free-viewpoint image is received by another information processing device that functions as the main controller for creating clips, and the information of the images before and after the connection with the free-viewpoint image is also transmitted and used for output image generation in the other information processing device. To be able to.
  • the transmission control unit controls to transmit information specifying the time length of the pre-image or the post-image connected to the generated free-viewpoint image to the other information processing apparatus. It is possible to do it. That is, the free-viewpoint image is received by another information processing device that functions as the main controller for creating the clip, and the time length information of the image before and after connecting to the free-viewpoint image is also transmitted to generate an output image in the other information processing device. To be able to be used for.
  • the information processing device that functions as the main controller for image creation sets a generation target image section for which a free viewpoint image is generated for a plurality of captured images simultaneously captured by a plurality of imaging devices.
  • the section specifying process to be specified the target image transmission control process for transmitting the image data of the generation target image section in each of the plurality of captured images as the image data used for generating the free viewpoint image in another information processing device, and the target image transmission control process.
  • an information processing device that functions as a free-viewpoint image server is a target for generating a free-viewpoint image in each of a plurality of captured images simultaneously captured by a plurality of imaging devices.
  • FIG. 1 shows a configuration example of an image processing system according to an embodiment of the present technology.
  • the image processing system includes an image creation controller 1, a free viewpoint image server 2, a video server 3, a plurality of (for example, 4) video servers 4A, 4B, 4C, 4D, NAS (Network Attached Storage) 5, a switcher 6, and image conversion. It has a unit 7, a utility server 8, and a plurality of (for example, 16) imaging devices 10.
  • the term “camera” refers to the imaging device 10.
  • “camera arrangement” means arrangement of a plurality of imaging devices 10.
  • video server 4 when the video servers 4A, 4B, 4C, and 4D are generically referred to without any particular distinction, they are referred to as "video server 4".
  • this image processing system based on the captured images (for example, image data V1 to V16) acquired from a plurality of imaging devices 10, a free viewpoint image corresponding to an observation image from an arbitrary viewpoint in a three-dimensional space is generated and freely selected. You can create an output clip that includes a viewpoint image.
  • connection state of each part is shown by a solid line, a broken line, and a double line.
  • the solid line indicates the connection of SDI (Serial Digital Interface), which is an interface standard for connecting broadcasting devices such as cameras and switchers, and is compatible with 4K, for example.
  • Image data is mainly transmitted and received between each device by SDI wiring.
  • the double line indicates the connection of the communication standard for constructing the computer network, for example, 10 Gigabit Ethernet.
  • the broken line between the video servers 3 and 4 indicates a state in which the video servers 3 and 4 equipped with the inter-server file sharing function are connected by, for example, a 10G network.
  • each video server can preview and send the material in the other video server. That is, a system using a plurality of video servers has been constructed so that efficient highlight editing and transmission can be realized.
  • Each image pickup device 10 is configured as a digital camera device having an image pickup element such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, and is an image captured as digital data (from image data V1). V16) is obtained. In this example, each image pickup device 10 obtains a captured image as a moving image.
  • an image pickup element such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor
  • each image pickup device 10 is supposed to capture a state in which a game such as basketball or soccer is being performed, and each of them is arranged in a predetermined direction at a predetermined position in a competition venue where the competition is held.
  • the number of image pickup devices 10 is 16, but the number of image pickup devices 10 may be at least 2 or more in order to enable the generation of a free viewpoint image.
  • FIG. 2 shows an example of arrangement of the image pickup device 10 around the basketball court. It is assumed that ⁇ is the imaging device 10. For example, this is an example of camera arrangement when it is desired to focus on the vicinity of the goal on the left side in the drawing. Of course, the camera arrangement and number are examples, and should be set according to the content and purpose of shooting and broadcasting.
  • the image creation controller 1 is composed of an information processing device.
  • the image creation controller 1 can be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like.
  • the image creation controller 1 performs control / operation management of the video servers 3 and 4 and processing for creating a clip.
  • the image creation controller 1 is a device that can be operated by the operator OP1.
  • the operator OP1 gives instructions such as selection of clip contents and creation.
  • the free-viewpoint image server 2 is configured as an information processing device that actually creates a free-viewpoint image (FV (Free View) clip, which will be described later) in response to an instruction from the image creation controller 1.
  • This free-viewpoint image server 2 can also be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like.
  • the free viewpoint image server 2 is a device that can be operated by the operator OP2.
  • the operator OP2 performs, for example, a camera work selection operation for creating an FV clip as a free viewpoint image.
  • the configuration and processing of the image creation controller 1 and the free viewpoint image server 2 will be described in detail later. Further, it is assumed that the operators OP1 and OP2 perform the operation.
  • the image creation controller 1 and the free viewpoint image server 2 may be arranged side by side and operated by one operator.
  • the video servers 3 and 4 are image recording devices, for example, a data recording unit such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and a control unit that controls data recording / playback of the data recording unit. To be equipped.
  • a data recording unit such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive)
  • HDD Hard Disk Drive
  • Each of the video servers 4A, 4B, 4C, and 4D is capable of inputting, for example, four systems, and simultaneously records images captured by four image pickup devices 10.
  • the video server 4A records the image data V1, V2, V3, V4.
  • the video server 4B records the image data V5, V6, V7, V8.
  • the video server 4C records the image data V9, V10, V11, and V12.
  • the video server 4D records the image data V13, V14, V15, V16.
  • the video servers 4A, 4B, 4C, and 4D are supposed to constantly record, for example, during a sports match to be broadcast.
  • the video server 3 is directly connected to, for example, the image creation controller 1, and can, for example, input two systems and output two systems.
  • Image data Vp and Vq are shown as two inputs.
  • the image data Vp and Vq it is possible to select an image captured by any of the two imaging devices 10 (any two of the image data V1 to V16). Of course, it may be an image captured by another imaging device.
  • the image data Vp and Vq can be displayed on the display by the image creation controller 1 as monitor images.
  • the operator OP1 can confirm the status of a scene shot / recorded for broadcasting, for example, by using the image data Vp and Vq input to the video server 3. Further, since the video servers 3 and 4 are connected to the file sharing state, the image creation controller 1 also monitors and displays the captured images of the imaging devices 10 recorded on the video servers 4A, 4B, 4C, and 4D. This allows the operator OP1 to check sequentially.
  • a time code is attached to the image captured by each imaging device 10, and it is possible to synchronize frames in the processing of the video servers 3, 4A, 4B, 4C, and 4D.
  • NAS5 is a storage device arranged on a network, and is composed of, for example, an SSD or an HDD. In the case of this example, NAS5 is when some frames of the image data V1, V2 ... V16 recorded on the video servers 4A, 4B, 4C, 4D are transferred for the generation of the free viewpoint image. , A device that stores a free-viewpoint image for processing in the free-viewpoint image server 2 or stores a created free-viewpoint image.
  • the switcher 6 is a device that inputs an image output via the video server 3 and finally selects and broadcasts the main line image PGMout.
  • a broadcasting director or the like performs necessary operations.
  • the image conversion unit 7 performs resolution conversion and composition of image data by, for example, the image pickup device 10, generates a monitoring image of the camera arrangement, and supplies the monitoring image to the utility server 8.
  • 16 systems of image data V1 to V16
  • V1 to V16 are converted into 4K images in resolution and then arranged in tiles to form 4 systems of images, which are supplied to the utility server 8.
  • the utility server 8 is a computer device capable of performing various related processes.
  • the utility server 8 is a device that detects camera movement for calibration.
  • the utility server 8 monitors the image data from the image conversion unit 7 and detects the movement of the camera.
  • the camera movement is, for example, the movement of one of the arrangement positions of the image pickup apparatus 10 arranged as shown in FIG.
  • the information on the arrangement position of the image pickup apparatus 10 is an important element for generating a free-viewpoint image, and if the arrangement position changes, it is necessary to redo the parameter setting. Therefore, the movement of the camera is monitored.
  • the image creation controller 1, the free-viewpoint image server 2, the video servers 3, 4, and the utility server 8 in the above configuration can be realized as an information processing device 70 having the configuration shown in FIG. 3, for example.
  • the CPU 71 of the information processing apparatus 70 executes various processes according to a program stored in the ROM 72 or a program loaded from the storage unit 79 into the RAM 73.
  • the RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes.
  • the CPU 71, ROM 72, and RAM 73 are connected to each other via a bus 74.
  • An input / output interface 75 is also connected to the bus 74.
  • An input unit 76 including an operator and an operation device is connected to the input / output interface 75.
  • various controls and operation devices such as a keyboard, mouse, keys, dial, touch panel, touch pad, and remote controller are assumed.
  • the user's operation is detected by the input unit 76, and the signal corresponding to the input operation is interpreted by the CPU 71.
  • a display unit 77 made of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel and an audio output unit 78 made of a speaker or the like are connected to the input / output interface 75 as one or a separate body.
  • the display unit 77 is a display unit that performs various displays, and is composed of, for example, a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, and the like.
  • the display unit 77 executes the display of various images for image processing, moving images to be processed, and the like on the display screen based on the instruction of the CPU 71. Further, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface) based on the instruction of the CPU 71.
  • GUI Graphic User Interface
  • a storage unit 79 composed of a hard disk, a solid-state memory, or the like, or a communication unit 80 composed of a modem or the like may be connected to the input / output interface 75.
  • the communication unit 80 performs communication processing via a transmission line such as the Internet, wire / wireless communication with various devices, bus communication, and the like.
  • a drive 82 is also connected to the input / output interface 75, if necessary, and a removable recording medium 81 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.
  • the drive 82 can read data files such as an image file MF and various computer programs from the removable recording medium 81.
  • the read data file is stored in the storage unit 79, and the image and sound included in the data file are output by the display unit 77 and the sound output unit 78. Further, the computer program or the like read from the removable recording medium 81 is installed in the storage unit 79 as needed.
  • software can be installed via network communication by the communication unit 80 or a removable recording medium 81.
  • the software may be stored in the ROM 72, the storage unit 79, or the like in advance.
  • FIG. 4 shows a section specifying processing unit 21, a target image transmission control unit 22, and an output image generation unit 23 as functions formed in the CPU 71 of the information processing device 70 that serves as the image creation controller 1.
  • the section specifying processing unit 21 performs a process of specifying a generation target image section to be generated as a free viewpoint image for a plurality of captured images (image data V1 to V16) simultaneously captured by the plurality of imaging devices 10. For example, when the operator OP1 performs an operation of selecting a scene to be replayed in an image, the time code for that scene, particularly the section of the scene to be a free viewpoint image (image section to be generated) can be specified. , Performs processing such as notifying the free viewpoint image server 2 of the time code.
  • the image section to be generated means a frame section that is actually a free viewpoint image.
  • a free viewpoint image is generated for a certain frame in a moving image
  • that one frame is a generation target image section.
  • the in / out points for the free viewpoint image have the same time code.
  • the plurality of frames are the generation target image sections.
  • the in / out points for the free viewpoint image have different time codes.
  • the target image transmission control unit 22 uses the image data of the generation target image section in each of the plurality of captured images, that is, one or more frames for the image data V1 to V16, to generate the free viewpoint image in the free viewpoint image server 2. Controls transmission as image data. Specifically, control is performed to transfer the image data as the image section to be generated from the video servers 4A, 4B, 4C, 4D to NAS5.
  • the output image generation unit 23 performs a process of generating an output image (output clip) including the received free viewpoint image (FV clip) generated by the free viewpoint image server 2.
  • the image creation controller 1 adds an FV clip, which is a virtual image generated by the free-viewpoint image server 2 by the processing of the output image generation unit 23, a front clip, which is an actual moving image at a time before that, and a later clip.
  • the rear clips which are the actual moving images at the time, are combined on the time axis to form an output clip. That is, the front clip + FV clip + rear clip is regarded as one output clip.
  • the front clip + FV clip may be used as one output clip.
  • the FV clip + rear clip may be used as one output clip.
  • the output clip of only the FV clip may be generated without combining the front clip and the rear clip.
  • the image creation controller 1 generates an output clip including the FV clip and outputs it to the switcher 6 so that it can be used for broadcasting.
  • FIG. 5 shows a target image acquisition unit 31, an image generation processing unit 32, and a transmission control unit 33 as functions formed in the CPU 71 of the information processing device 70 which is the free viewpoint image server 2.
  • the target image acquisition unit 31 acquires the image data of the generation target image section that is the generation target of the free viewpoint image in each of the plurality of captured images (image data V1 to V16) simultaneously captured by the plurality of imaging devices 10. Perform the processing to be performed. That is, the image data of one frame or a plurality of frames specified by the in / out points of the generation target image section specified by the image creation controller 1 by the function of the section specifying processing unit 21 is output from the video servers 4A, 4B, 4C, 4D to NAS5. It can be acquired via and used for the generation of a free viewpoint image.
  • the target image acquisition unit 31 acquires image data of one frame or a plurality of frames of the generation target image section for all of the image data V1 to V16.
  • the reason for acquiring the image data of the image section to be generated for all of the image data V1 to V16 is to generate a high-quality free-viewpoint image.
  • the image generation processing unit 32 is a function of generating a free viewpoint image, that is, an FV clip in the case of this example, using the image data acquired by the target image acquisition unit 31.
  • the image generation processing unit 32 performs processing such as modeling processing including 3D model generation and subject analysis, and rendering to generate a free viewpoint image which is a two-dimensional image from the 3D model.
  • the 3D model generation represents a subject in a three-dimensional space (that is, from a two-dimensional image) based on an image captured by each imaging device 10 and a camera parameter for each imaging device 10 input from, for example, a utility server 8.
  • This is a process for generating 3D model data (which restores the three-dimensional structure of the subject).
  • the 3D model data includes data in which the subject is represented by a (X, Y, Z) three-dimensional coordinate system.
  • Subject analysis analyzes the position, orientation, and posture of a subject as a person (player) based on 3D model data. Specifically, the position of the subject is estimated, a simple model of the subject is generated, and the orientation of the subject is estimated. Then, a free viewpoint image is generated based on the 3D model data and the subject analysis information. For example, a free viewpoint image that moves the viewpoint is generated for a 3D model in which the player who is the subject is stationary.
  • FIG. 6A shows an image of a free viewpoint image in which a subject is captured from a required viewpoint set in a three-dimensional space.
  • the subject HS1 is viewed substantially in front and the subject HS2 is viewed substantially in the rear.
  • FIG. 6B shows an image of a virtual viewpoint image when a viewpoint is set in which the position of the viewpoint is changed in the direction of the arrow C in FIG. 6A and the subject HS1 is viewed substantially backward.
  • the subject HS2 is viewed substantially in front, and the subject HS3 and the basket goal, which were not projected in FIG. 6A, are projected.
  • the viewpoint is gradually moved in the direction of arrow C, and an image of about 1 to 2 seconds that reaches the state of FIG. 6B is generated as a free viewpoint image (FV clip). ..
  • FV clip free viewpoint image
  • the time length of the FV clip as a free viewpoint image and the trajectory of the viewpoint movement can be considered in various ways.
  • the transmission control unit 33 controls to transmit the free viewpoint image (FV clip) generated by the image generation processing unit 32 to the image creation controller 1 via the NAS 5 as described above.
  • the transmission control unit 33 controls to transmit the accompanying information for generating the output image to the image creation controller 1.
  • the accompanying information is assumed to be information that specifies the image of the front clip or the rear clip. That is, it is information for designating which image of the image data V1 to V16 is used to create (cut out) the front clip or the rear clip.
  • information that specifies the time length of the front clip and the rear clip is also assumed as incidental information.
  • GUI> As described above, the FV clip as the free viewpoint image is created by the free viewpoint image server 2 in response to the operation of the operator OP2.
  • the display unit 77 displays the GUI screen 40 as shown in FIG. 7, and the operator OP2 can confirm and operate the GUI screen 40.
  • the image of the image section to be generated is displayed on the monitor so that the operator OP2 can confirm the content of the scene for generating the free viewpoint image.
  • the scene list display unit 42 displays, for example, a list of scenes designated for the image section to be generated.
  • the operator OP2 can select the scene to be displayed in the scene window 41 on the scene list display unit 42.
  • Camera work is information that mainly indicates the movement locus of a viewpoint in a free viewpoint image. For example, when creating an FV clip that changes the direction and distance of the image pickup device 10 as the position (that is, the viewpoint) of the subject for which the 3D model is generated, the necessary parameters for forming the locus of the viewpoint. However, it is used as camera work information.
  • the display of the camera work for example, the trajectory of the viewpoint movement is displayed.
  • the camera work list display unit 44 displays a list of various camera work information created and stored in advance.
  • the operator OP2 can select the camera work to be used for FV clip generation from the camera works displayed on the camera work list display unit 44.
  • Various parameters related to the selected camera work are displayed on the parameter display unit 45.
  • Information about transmitting the created FV clip to the image creation controller 1 is displayed in the transmission window 46.
  • FIG. 8 shows a state in which the output clip is configured by connecting the front clip, the FV clip, and the rear clip as an example.
  • the front clip is an actual moving image of the section of the time code TC1 to TC2 in a certain image data Vx of the image data V1 to the image data V16.
  • the rear clip is an actual moving image of the section of the time code TC5 to TC6 in a certain image data Vy of the image data V1 to the image data V16. It is usually assumed that the image data Vx is the image data of the image pickup device 10 before the start of the viewpoint movement by the FV clip, and the image data Vy is the image data of the image pickup device 10 at the end of the viewpoint movement by the FV clip.
  • the front clip is a video with a time length of t1
  • the FV clip is a free viewpoint image with a time length of t2
  • the rear clip is a video with a time length of t3.
  • the playback time length of the entire output clip is t1 + t2 + t3.
  • a configuration such as a 1.5-second moving image, a 2-second free-viewpoint image, a 1.5-second moving image, or the like can be considered.
  • the FV clip for moving the viewpoint while the video time is stopped is called “still image FV clip”
  • the FV clip for moving the viewpoint without stopping the video time is called "video FV clip". I will do it.
  • FIG. 9 shows the still image FV clip with reference to the frame of the moving image.
  • the time codes TC1 and TC2 of the front clip are the time codes of the frames F1 and F81
  • the time codes TC5 and TC6 of the rear clip become the time codes of the frames F83 and F166. That is, it is a case of generating a free viewpoint image in which the viewpoint moves with respect to the still image of one frame of the frame F82.
  • the moving image FV clip is as shown in FIG.
  • the time codes TC1 and TC2 of the front clip are the time codes of the frames F1 and F101
  • the time codes of the frames F102 and F302 are the time codes TC3 and TC4 of FIG.
  • the time codes TC5 and TC6 of the rear clip become the time codes of the frames F303 and F503. That is, it is a case where a free viewpoint image in which the viewpoint moves is generated for a moving image in a section of a plurality of frames from the frame F102 to the frame 302.
  • the generation target image section determined by the image creation controller 1 is one frame section of the frame F82 when the still image FV clip of FIG. 9 is created, and the frame when the moving image FV clip of FIG. 10 is created. It is a section of a plurality of frames from F102 to frame 302.
  • FIG. 9 an example of the image content of the output clip is shown in FIG.
  • the front clip is an actual moving image from frame F1 to frame F81.
  • the FV clip it becomes a virtual image in which the viewpoint is moved in the scene of the frame F81.
  • the rear clip is an actual moving image from frame F83 to frame F166.
  • an output clip including an FV clip is generated in this way and used as an image to be broadcast.
  • the in / out points of the FV clip and the images between them may include both the image of the viewpoint from the actual camera position and the image of the viewpoint from a place other than the actual camera position, and are actually It may be composed of a plurality of images of the viewpoint from the camera position of the camera, or may be composed of a plurality of images of the viewpoint from a place other than the actual camera position.
  • Step S1 Scene selection
  • the operator OP1 first selects a scene to be used as an FV clip. For example, the operator OP1 searches for a scene to be used as an FV clip while monitoring the captured image displayed on the display unit 77 on the image creation controller 1 side. Then, the generation target image section of one frame or a plurality of frames is selected.
  • the information of the image section to be generated is transmitted to the free viewpoint image server 2, and the operator OP2 can be recognized by the GUI on the display unit 77 on the free viewpoint image server 2 side.
  • Step S2 Scene image transfer instruction
  • the operator OP2 operates the image transfer instruction of the corresponding scene according to the designation of the generation target image section.
  • the free-viewpoint image server 2 transmits a transfer request for image data in the sections of the time codes TC3 and TC4 to the image creation controller 1.
  • Step S3 Synchronous cutout
  • the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the image data transfer request, and for each of the 16 systems of image data from the image data V1 to the image data V16. , The section of the time codes TC3 and TC4 is cut out.
  • Step S4 NAS transfer Then, the image creation controller 1 transfers the data of all the time codes TC3 and TC4 sections of the image data V1 to the image data V16 to the NAS5.
  • Step S5 Thumbnail display
  • the free-viewpoint image server 2 displays thumbnails of image data V16 from image data V1 in the sections of time codes TC3 and TC4 transferred to NAS5.
  • Step S6 Scene check
  • the operator OP2 confirms the scene contents of the section indicated by the time codes TC3 and TC4 on the GUI screen 40 by the free viewpoint image server 2.
  • Step S7 Camera work selection
  • the operator OP2 selects the camera work that is considered appropriate on the GUI screen 40 according to the scene content.
  • Step S8 Generation execution The operator OP2 performs an operation of generating and executing an FV clip after selecting the camera work.
  • Step S9 Modeling
  • the free-viewpoint image server 2 uses parameters such as frame data in the sections of time codes TC3 and TC4 in each of the image data V1 to V16, and the arrangement position of each image pickup device 10 input in advance. Then, a 3D model of the subject is generated, the subject is analyzed, and the like.
  • the free viewpoint image server 2 generates a free viewpoint image based on the 3D model data and the subject analysis information. At this time, a free viewpoint image is generated so that the viewpoint is moved based on the camera work selected in step S7.
  • Step S11 Transfer The free viewpoint image server 2 transfers the generated FV clip to the image creation controller 1. At this time, not only the FV clip but also the specification information of the front clip and the rear clip and the specification information of the time length of the front clip and the rear clip can be transmitted as incidental information.
  • Step S13 Playlist generation
  • the image creation controller 1 generates an output clip using the transmitted FV clip.
  • one or both of the front clip and the rear clip are combined with the FV clip on the time axis to generate an output clip.
  • This output clip may be generated as stream data in which each frame as a front clip, each virtually generated frame as an FV clip, and each frame as a rear clip are actually connected in chronological order.
  • the output clip can be played back without generating the stream data.
  • Step S14 Quality confirmation
  • the GUI on the image creation controller 1 side performs playback based on the playlist, and the operator OP1 confirms the contents of the output clip.
  • Step S15 Reproduction instruction
  • the operator OP1 gives a reproduction instruction by a predetermined operation according to the quality confirmation.
  • the image creation controller 1 recognizes the input of the playback instruction.
  • Step S16 Reproduction The image creation controller 1 supplies the output clip to the switcher 6 in response to the reproduction / reproduction instruction. This makes it possible to broadcast the output clip.
  • FIG. 13 shows the processing of the image creation controller 1 and FIG. 14 shows the processing of the free-viewpoint image server 2 for realizing the above processing procedure.
  • FIG. 13 is a process executed by the CPU 71 of the information processing apparatus 70 as the image creation controller 1 by each function shown in FIG.
  • FIG. 14 is a process executed by the CPU 71 of the information processing apparatus 70, which is the free-viewpoint image server 2 by each function shown in FIG.
  • step S101 of FIG. 13 the image creation controller 1 controls the captured image display.
  • the image data Vp and Vq shown in FIG. 1 are displayed on the display unit 77 so that they can be monitored by the operator OP1.
  • step S102 of FIG. 13 the image creation controller 1 monitors that the selection operation of the image section to be generated has been performed. If the selection operation has not been performed, the process shifts to another process.
  • step S102 When the operator OP1 detects the selection operation of the image section to be generated at a certain point, the image creation controller 1 proceeds from step S102 to step S103 to generate information on the image section to be generated, that is, an FV clip, for the free viewpoint image server 2. Performs processing to notify the information of the in point / out point indicating the section of the scene to be performed. Then, in step S104, the image creation controller 1 waits for a transfer instruction from the free viewpoint image server 2.
  • the notification of the image section to be generated from the image creation controller 1 is checked in step S201 of FIG.
  • the free viewpoint image server 2 proceeds from step S201 to step S202 to perform a process for requesting the transfer of image data.
  • the free-viewpoint image server 2 displays a notification of the image section to be generated on the GUI screen and presents it to the operator OP2 to enable a transfer request operation.
  • a transfer request is transmitted to the image creation controller 1 in response to the operation of the operator OP2.
  • the transfer request may be performed without requiring any particular operation.
  • the free viewpoint image server 2 After transmitting the transfer request, the free viewpoint image server 2 waits in step S203 for the data of the frame of the required image section to be generated in the image data (V1 to V16) to be transferred to NAS5.
  • the image creation controller 1 When the image creation controller 1 receives the transfer request from the free viewpoint image server 2, the image creation controller 1 proceeds from step S104 to step S105 in FIG. 13 to cut out the image section to be generated for the video servers 4A, 4B, 4C, and 4D. Controls to execute the transfer to NAS5. As a result, the frames (one frame or a plurality of frames) of the in-point / out-point sections of the image data section to be generated in the image data V1 to V16 are transferred to the NAS 5. Then, the image creation controller 1 proceeds to step S106 and waits for the FV clip to be transmitted from the free viewpoint image server 2.
  • the free-viewpoint image server 2 can acquire the image data in a processable state by transferring the image data to the NAS 5.
  • the process proceeds from step S203 to step S204 in FIG. 14, and the acquired image data is used on the GUI screen 40. Display the scene. This allows the operator OP2 to confirm the content of the scene for creating the FV clip, and the free viewpoint image server 2 monitors the camera work selection operation by the operator OP2 in step S205.
  • the free viewpoint image server 2 proceeds from step S205 to step S206 to determine the camera work and set its parameters for processing. Then, in step S207, the creation execution operation by the operator OP2 is monitored. Of course, in practice, it is desirable that the operator OP2 can redo the camera work selection.
  • the free viewpoint image server 2 proceeds from step S207 to step S208 to execute the modeling process described above. Then, rendering is performed in step S209 to generate an FV clip. In step S210, the free viewpoint image server 2 displays the generated FV clip on the GUI screen 40 so that the operator OP2 can confirm it. Then, the transmission operation by the operator OP2 is monitored in step S211 and the FV clip is transmitted to the image creation controller 1 in step S212 according to the transmission operation. At this time, designated information of the front clip and the rear clip, and designated information of the time length of the front clip and the rear clip may be transmitted.
  • step S107 to set the front clip and the rear clip. For example, select a front clip or a back clip based on accompanying information. Specifically, which of the image data V1 to V16 is used for the front clip and the rear clip is set.
  • the front clip is preferably an image taken by the image pickup device 10 that matches the viewpoint of the start image of the FV clip.
  • the rear clip is preferably an image obtained by the imaging device 10 that matches the viewpoint of the end image of the FV clip.
  • Which image of the imaging device 10 is used as the viewpoint position of the start image and the viewpoint position of the end image of the FV clip is information grasped by the free viewpoint image server 2. That is, it is the information of the image pickup device 10 for the start image of the FV clip and the image pickup device 10 for the end image defined by the camera work used in the FV clip generation process this time. Therefore, by making the information for specifying the front clip and the back clip transmitted from the free viewpoint image server 2 as the information indicating these as incidental information, the image creation controller 1 can easily perform the front clip and the back clip. Can be set.
  • the image creation controller 1 also sets the time lengths of the front clip and the rear clip, which may be set according to the incidental information to be transmitted. For example, when the operator OP2 wants to determine the length of the FV clip, the time length, and the length of the front and rear clips according to the camera work, the time length of the front clip and the rear clip is determined on the free viewpoint image server 2 side, and the image creation controller. For 1, it is desirable to set the front clip and the rear clip according to it. Of course, the image creation controller 1 may determine the time length of the front clip and the rear clip, or may determine a fixed time length.
  • step S108 the image creation controller 1 creates an output clip by combining the front clip, the FV clip, and the rear clip.
  • an output clip can be created by generating playlists and virtually combining them.
  • step S109 the image creation controller 1 performs a process of transferring the output clip to the switcher 6.
  • the output clip is reproduced according to the playlist so that the operator OP1 can confirm the contents, and the output clip is transmitted to the switcher 6 according to the operation of the operator OP1.
  • the image creation controller 1 and the free-viewpoint image server 2 perform the processes shown in FIGS. 13 and 14, so that an output clip including the FV clip is generated and broadcast.
  • Step S30 HD output
  • the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to output image data to the image conversion unit 7 for camera movement detection.
  • the images from the video servers 4A, 4B, 4C, and 4D, that is, the images of the 16 image pickup devices 10 are converted in resolution by the image conversion unit 7 and supplied to the utility server 8.
  • Step S31 The background generation utility server 8 generates a background image based on the supplied image. Since the background image is an image that does not change unless the camera position is changed, for example, a background image excluding a subject such as a player is generated for 16 systems of image data (V1 to V16). Step S32: Difference confirmation The background image is displayed in GUI, so that the operator OP2 can confirm the change in the image. Step S33: Automatic movement detection By comparing the background images at each time point, the camera movement can be automatically detected.
  • Step S34 Camera movement detection As a result of the above step S32 or step S33, the movement of a certain imaging device 10 is detected.
  • -Step S35 Image acquisition Calibration is required according to the movement of the image pickup apparatus. Therefore, the utility server 8 requests the image creation controller 1 for the image data in the state after the movement.
  • Step S36 Clip cutting The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the request for image acquisition from the utility server 8 to execute clip cutting for the image data V1 to V16. ..
  • Step S37 NAS transfer The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as a clip to the NAS 5.
  • Step S38 Feature point correction
  • the utility server 8 can refer to and display the image of the state after the camera is moved.
  • the operator OP2 performs operations necessary for calibration such as correction of feature points.
  • Step S40 Background reacquisition
  • the utility server 8 requests reacquisition of image data for the background image in response to the operation of the operator OP2 after the calibration.
  • Step S41 Clip cutting
  • the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the request for image acquisition from the utility server 8 to execute clip cutting for the image data V1 to V16. ..
  • Step S42 NAS transfer
  • the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as a clip to the NAS 5.
  • Step S43 The background generation utility server 8 generates a background image using the image data transferred to the NAS 5. This is, for example, a background image that serves as a reference for subsequent camera movement detection.
  • the information processing device 70 as the image creation controller 1 of the embodiment is a generation target for generating a free viewpoint image for a plurality of captured images (for example, image data V1 to V16) simultaneously captured by the plurality of imaging devices 10.
  • Image data of the section specifying processing unit 21 that performs the process of specifying the image section and the image data of the image section to be generated in each of the plurality of captured images (V1 to V16) are used for generating the FV clip in the free viewpoint image server 2. It is provided with a target image transmission control unit 22 that controls transmission as an image, and an output image generation unit 23 that generates an output clip including a received FV clip.
  • the information processing device 70 as the free viewpoint image server 2 of the embodiment is a target for generating a free viewpoint image in each of a plurality of captured images (V1 to V16) simultaneously captured by the plurality of imaging devices 10.
  • the target image acquisition unit 31 that acquires the image data of the generation target image section, the image generation processing unit 32 that generates a free viewpoint image using the acquired image data, and the generated free viewpoint image are transmitted to the image creation controller 1.
  • a transmission control unit 33 is provided.
  • the image creation controller 1 transfers only the image data of the image section to be generated of the FV clip to the NAS 5.
  • the free-viewpoint image server 2 generates an FV clip based on the image data of 16 systems transferred to the NAS 5, and transmits the FV clip to the image creation controller 1.
  • the image creation controller 1 generates an output clip including an FV clip. Therefore, the amount of image data transferred to NAS5 (free viewpoint image server 2 side) is small, the processing load on the free viewpoint image server 2 side is small, and the FV clip can be generated in a short time. Further, it suffices if the FV clip can be transmitted to the image creation controller 1 side, which reduces the processing load of the free-viewpoint image server 2 and is effective for quick clip creation. Normally, when creating an output clip including an FV clip, for example, an image of the entire scene of the output clip is transmitted to NAS5 (free viewpoint image server 2 side), and a front clip or a rear clip is sent on the free viewpoint image server 2 side.
  • the process of generating the entire output clip including the above is performed, and a relatively long processing time is required.
  • the free viewpoint image server 2 may generate an FV clip from the image data of the scene in the section of the FV clip.
  • the operation of the embodiment becomes extremely useful for shortening the output clip generation time. For example, in the case of live broadcasting such as sports, it may be required to create an output clip in about 10 seconds in order to broadcast an immediate replay. It is also advantageous for such applications.
  • the image creation controller 1 of the embodiment performs a process of notifying the free viewpoint image server 2 of the information of the specified generation target image section, and also receives the image data of the generation target image section in response to the request from the free viewpoint image server 2. Controls the transmission of. Further, the free viewpoint image server 2 of the embodiment requests transmission of a plurality of image data of the generation target image section in each of the plurality of captured images in response to the notification of the generation target image section. Thereby, after the image creation controller 1 notifies the target image section, the generation of the FV clip can be executed according to the convenience of the free viewpoint image server 2 or the operator OP2 side. As a result, the processing is not complicated on the free viewpoint image server 2 side, and the FV clip generation processing and the operation of the operator OP2 can be facilitated.
  • the section specifying processing unit 21 of the image creation controller 1 gives an example of specifying a section of one frame of the captured image as the image section to be generated.
  • the target image acquisition unit 31 of the free-viewpoint image server 2 acquires the image data of one frame set as the generation target image section in each of the plurality of captured images. That is, it is a case where a still image FV clip is generated.
  • the free viewpoint image server 2 has one frame for each of the image data V1 to the image data V16 by the 16 image pickup devices 10. You only have to get it. This makes it possible to extremely reduce the processing load in the transfer processing to the NAS 5 and the processing of the free-viewpoint image server 2.
  • the section specifying processing unit 21 of the image creation controller 1 gives an example of specifying a section of a plurality of frames of the captured image as a generation target image section.
  • the target image acquisition unit 31 of the free-viewpoint image server 2 acquires image data of a section of a plurality of frames as a generation target image section in each of the plurality of captured images. That is, it is a case of generating a moving image FV clip.
  • the free viewpoint image server 2 sets the FV clip for the image data V1 to the image data V16 by the 16 image pickup devices 10, respectively. It is only necessary to acquire a plurality of frames of the video section to be played. Since the acquisition of the minimum necessary frame section, no useless processing occurs in the transfer processing to the NAS 5 and the processing of the free viewpoint image server 2.
  • the output image generation unit 23 of the image creation controller 1 generates an output image in which the front image or the rear image is combined with the free viewpoint image on the time axis. That is, the output image generation unit 23 can generate an output clip in which the front clip or the rear clip is combined with the received FV clip, for example, to generate an output clip in which the viewpoint is moved in the middle of the scene. Then, by combining the front clip and the rear clip with respect to the FV clip on the image creation controller 1, the processing load of the free viewpoint image server 2 is reduced, which is effective for quick clip creation.
  • the output image generation unit 23 of the image creation controller 1 generates an output image as a virtual clip in which the free viewpoint image and the front image or the rear image are virtually combined on the time axis by playlist information. bottom. That is, by forming a playlist in which the front clip and the rear clip are connected to the FV clip, the output clip can be played along with the playlist without actually creating an output clip as an image data stream. Image output is possible. This simplifies the processing on the image creation controller 1 side and shortens the output clip creation time, which is suitable for requesting quick output clip creation.
  • the transmission control unit 33 of the free-viewpoint image server 2 controls to transmit to the image creation controller 1 information that specifies a front image or a back image to be connected to the generated free-viewpoint image as incidental information.
  • the output image generation unit 23 of the image creation controller 1 acquires information for designating the front image or the rear image to be connected to the free viewpoint image from the free viewpoint image server 2, and based on the information, the front image or the rear image is added to the free viewpoint image. It was assumed that the output image was generated by combining the post-images on the time axis.
  • the FV clip is an image that changes from a certain viewpoint, that is, an image of a certain image pickup device 10, to another viewpoint, that is, an image of another image pickup device 10.
  • the image creation controller 1 can easily prepare an appropriate image as a front clip or a back clip connected to the FV clip by acquiring the information of the front clip and the back clip from the free viewpoint image server 2. Become. Then, by combining them on the time axis, it is possible to generate an output clip having good compatibility before and after the viewpoint movement. Further, as the free viewpoint image server 2, since the front clip and the rear clip can be specified from the camera work information, the processing load does not increase.
  • the transmission control unit 33 of the free viewpoint image server 2 transmits, as incidental information, information for designating the time length of the front image or the back image connected to the generated free viewpoint image to the image creation controller 1. I gave an example of doing. Then, the output image generation unit 23 of the image creation controller 1 acquires information for specifying the time length of the front image or the rear image to be connected to the free viewpoint image from the free viewpoint image server 2, and based on the information, the front image or the rear image. An image was prepared, and an output image was generated by combining a front image or a rear image with a free viewpoint image on the time axis.
  • the operator OP2 that creates an FV clip can set the time length of the output clip according to the contents of the FV clip.
  • the image generation processing unit 32 generates a free viewpoint image using the camera work information selected from the camera work information (viewpoint movement locus information) stored in advance. Said.
  • a plurality of camera work information is stored as information indicating the trajectory of the viewpoint movement, and the operator OP2 selects the camera work information according to the designated scene.
  • the FV clip is generated only by the operator OP2 selecting the camera work information according to the content of the scene as the image section to be generated. Therefore, it is possible to create an FV clip of viewpoint movement (camera work) according to the scene content in a short time.
  • a program for causing, for example, a CPU, a DSP, or a device including these to execute the process described with reference to FIG. 13 can be considered. That is, the program of the embodiment is a section specifying process (S102, S103) and the target image transmission control process (S105) for transmitting the image data of the generation target image section in each of the plurality of captured images as the image data used for generating the free viewpoint image in the other information processing apparatus.
  • This is a program that causes an information processing apparatus to execute an output image generation process (S107, S108) that generates an output image including a free viewpoint image.
  • the above-mentioned image creation controller 1 can be realized in a device as an information processing device 70.
  • a program for causing, for example, a CPU, a DSP, or a device including these to execute the process described with reference to FIG. 14 can be considered. That is, the program of the embodiment acquires the image data of the generation target image section that is the generation target of the free viewpoint image in each of the plurality of captured images (V1 to V16) simultaneously captured by the plurality of imaging devices 10. Image data acquisition processing (S202, S203), image generation processing (S204 to S209) that generates a free-viewpoint image using the image data acquired in the image data acquisition processing, and other information processing devices that generate the generated free-viewpoint image.
  • the above-mentioned free-viewpoint image server 2 can be realized in a device as an information processing device 70.
  • These programs can be recorded in advance in an HDD as a recording medium built in a device such as a computer device, a ROM in a microcomputer having a CPU, or the like.
  • a recording medium built in a device such as a computer device, a ROM in a microcomputer having a CPU, or the like.
  • flexible discs CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) discs, DVDs (Digital Versatile Discs), Blu-ray discs (Blu-ray Discs (registered trademarks)), magnetic discs, semiconductor memories, It can be temporarily or permanently stored (recorded) on a removable recording medium such as a memory card.
  • a removable recording medium can be provided as so-called package software.
  • it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
  • LAN Local Area Network
  • the image creation controller 1 and the free viewpoint image server 2 of the embodiment it is suitable for a wide range of provision of the image creation controller 1 and the free viewpoint image server 2 of the embodiment.
  • a program for example, by downloading a program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can be transferred to the image creation controller 1 of the present disclosure or a free viewpoint. It can function as an image server 2.
  • the present technology can also adopt the following configurations.
  • a section identification processing unit that performs processing to specify a generation target image section to be generated as a free viewpoint image for a plurality of captured images simultaneously captured by a plurality of imaging devices.
  • a target image transmission control unit that controls transmission of image data of the generation target image section in each of the plurality of captured images as image data used for generation of a free viewpoint image in another information processing device.
  • An information processing device including an output image generator that generates an output image including a received free-viewpoint image.
  • the section specifying processing unit performs a process of notifying the other information processing apparatus of the information of the specified image section to be generated.
  • the information processing device wherein the target image transmission control unit controls transmission of image data of the generation target image section in response to a request from the other information processing device.
  • the section specifying processing unit specifies a section of one frame of a captured image as the image section to be generated.
  • the section specifying processing unit specifies a section of a plurality of frames of an captured image as the image section to be generated.
  • the output image generation unit generates an output image in which a front image or a rear image is combined with a free viewpoint image on the time axis.
  • the output image generation unit generates an output image as a virtual clip in which the free viewpoint image and the front image or the rear image are virtually combined on the time axis based on the playlist information.
  • the output image generation unit acquires information for designating a front image or a back image to be connected to the free viewpoint image from the other information processing apparatus, and based on the information, sets the front image or the back image on the free viewpoint image on the time axis.
  • the information processing apparatus according to any one of (1) to (6) above, which generates an output image combined above.
  • the output image generation unit acquires information for specifying the time length of the front image or the rear image to be connected to the free viewpoint image from the other information processing apparatus, prepares the front image or the rear image based on the information, and prepares the front image or the rear image.
  • the information processing apparatus according to any one of (1) to (7) above, which generates an output image in which a front image or a rear image is combined with a free viewpoint image on the time axis.
  • a target image acquisition unit that acquires image data of a generation target image section that is a generation target of a free viewpoint image in each of a plurality of captured images simultaneously captured by a plurality of imaging devices.
  • An image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit, and an image generation processing unit.
  • An information processing device including a transmission control unit that controls transmission of the generated free-viewpoint image to another information processing device.
  • the target image acquisition unit requests transmission of a plurality of image data of the generation target image section in each of the plurality of captured images in response to the notification of the generation target image section. Processing equipment.
  • the target image acquisition unit acquires image data of one frame set as the generation target image section in each of a plurality of captured images.
  • the information processing device (9) or (10) above, wherein the target image acquisition unit acquires image data of a section of a plurality of frames as the generation target image section in each of the plurality of captured images.
  • the image generation processing unit The information processing apparatus according to any one of (9) to (12) above, which generates a free viewpoint image using the viewpoint movement locus information selected from the viewpoint movement locus information stored in advance.
  • the transmission control unit The information processing device according to any one of (9) to (13) above, which controls transmission of information specifying a front image or a rear image to be connected to the generated free viewpoint image to the other information processing device.
  • the transmission control unit The information processing apparatus according to any one of (9) to (14) above, which controls transmission of information specifying the time length of the pre-image or post-image connected to the generated free-viewpoint image to the other information processing apparatus. .. (16) As an image processing system that has a first information processing device and a second information processing device and outputs an output image including a free viewpoint image.
  • the first information processing device is A section identification processing unit that performs processing to specify a generation target image section to be generated as a free viewpoint image for a plurality of captured images simultaneously captured by a plurality of imaging devices.
  • a target image transmission control unit that controls transmission of image data of the generation target image section in each of the plurality of captured images as image data used for generation of a free viewpoint image in the second information processing apparatus.
  • An output image generator that generates an output image including the received free-viewpoint image
  • the second information processing device is A target image acquisition unit for acquiring image data of the generation target image section for which a free viewpoint image is generated in each of the plurality of captured images, and a target image acquisition unit.
  • An image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit, and an image generation processing unit.
  • a transmission control unit that controls transmission of the generated free-viewpoint image to the first information processing device, and An image processing system equipped with.
  • (17) Information processing device For a plurality of captured images simultaneously captured by a plurality of imaging devices, a section specifying process for specifying a generation target image section for which a free viewpoint image is to be generated, and a section specifying process.
  • Target image transmission control processing for transmitting the image data of the generation target image section in each of the plurality of captured images as image data used for generating a free viewpoint image in another information processing device.
  • Output image generation processing that generates an output image including the received free viewpoint image, Information processing method to perform.
  • Information processing device Image data acquisition processing for acquiring image data of a generation target image section for which a free viewpoint image is to be generated in each of a plurality of captured images simultaneously captured by a plurality of imaging devices. An image generation process for generating a free viewpoint image using the image data acquired in the image data acquisition process, and an image generation process. Transmission control processing that controls the transmission of the generated free-viewpoint image to other information processing devices, Information processing method to perform.
  • Image creation controller Free viewpoint image server 3, 4, 4A, 4B, 4C, 4D video server 5 NAS 6 Switcher 7 Image conversion unit 8 Utility server 10 Imaging device 21 Section identification processing unit 22 Target image transmission control unit 23 Output image generation unit 31 Target image acquisition unit 32 Image generation processing unit 33 Transmission control unit 70 Information processing device 71 CPU 77 Display

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Processing Or Creating Images (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
PCT/JP2021/005519 2020-03-30 2021-02-15 情報処理装置、画像処理システム、情報処理方法 Ceased WO2021199735A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP21781577.8A EP4113453A4 (en) 2020-03-30 2021-02-15 INFORMATION PROCESSING DEVICE, IMAGE PROCESSING SYSTEM AND INFORMATION PROCESSING METHOD
JP2022511634A JP7635778B2 (ja) 2020-03-30 2021-02-15 情報処理装置、画像処理システム、情報処理方法
US17/907,854 US12219122B2 (en) 2020-03-30 2021-02-15 Information processing apparatus, image processing system, and information processing method
CN202180023809.4A CN115315730A (zh) 2020-03-30 2021-02-15 信息处理装置、图像处理系统和信息处理方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020061248 2020-03-30
JP2020-061248 2020-03-30

Publications (1)

Publication Number Publication Date
WO2021199735A1 true WO2021199735A1 (ja) 2021-10-07

Family

ID=77928311

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/005519 Ceased WO2021199735A1 (ja) 2020-03-30 2021-02-15 情報処理装置、画像処理システム、情報処理方法

Country Status (5)

Country Link
US (1) US12219122B2 (https=)
EP (1) EP4113453A4 (https=)
JP (1) JP7635778B2 (https=)
CN (1) CN115315730A (https=)
WO (1) WO2021199735A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115379126B (zh) * 2022-10-27 2023-03-31 荣耀终端有限公司 一种摄像头切换方法及相关电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018030206A1 (ja) 2016-08-10 2018-02-15 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ カメラワーク生成方法及び映像処理装置
JP2019079298A (ja) * 2017-10-25 2019-05-23 キヤノン株式会社 情報処理装置、表示制御方法、及び、プログラム
WO2020012748A1 (ja) * 2018-07-12 2020-01-16 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム
JP2020036116A (ja) * 2018-08-28 2020-03-05 キヤノン株式会社 情報処理装置、情報処理装置の制御方法、及びプログラム

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7751647B2 (en) * 2005-12-08 2010-07-06 Lenel Systems International, Inc. System and method for detecting an invalid camera in video surveillance
JP4798358B2 (ja) * 2006-03-14 2011-10-19 セイコーエプソン株式会社 画像処理システム、表示装置および画像処理方法
US20110043689A1 (en) * 2009-08-18 2011-02-24 Wesley Kenneth Cobb Field-of-view change detection
EP2413286A1 (en) * 2010-07-29 2012-02-01 LiberoVision AG Image processing method and device for instant replay
EP2775731A1 (en) * 2013-03-05 2014-09-10 British Telecommunications public limited company Provision of video data
JP6299492B2 (ja) * 2014-07-03 2018-03-28 ソニー株式会社 情報処理装置、情報処理方法、およびプログラム
US20170244894A1 (en) * 2016-02-22 2017-08-24 Seastar Labs, Inc. Method and Apparatus for Managing Latency of Remote Video Production
JP6429829B2 (ja) 2016-05-25 2018-11-28 キヤノン株式会社 画像処理システム、画像処理装置、制御方法、及び、プログラム
CN110463182B (zh) * 2017-03-31 2021-07-13 松下知识产权经营株式会社 摄像系统以及校正方法
EP3606038B1 (en) * 2017-03-31 2021-12-08 Panasonic Intellectual Property Management Co., Ltd. Imaging system and correction method
JP7030452B2 (ja) * 2017-08-30 2022-03-07 キヤノン株式会社 情報処理装置、情報処理装置の制御方法、情報処理システム及びプログラム
JP2019197340A (ja) * 2018-05-09 2019-11-14 キヤノン株式会社 情報処理装置、情報処理方法、及び、プログラム
JP7207913B2 (ja) * 2018-09-07 2023-01-18 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018030206A1 (ja) 2016-08-10 2018-02-15 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ カメラワーク生成方法及び映像処理装置
JP2019079298A (ja) * 2017-10-25 2019-05-23 キヤノン株式会社 情報処理装置、表示制御方法、及び、プログラム
WO2020012748A1 (ja) * 2018-07-12 2020-01-16 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム
JP2020036116A (ja) * 2018-08-28 2020-03-05 キヤノン株式会社 情報処理装置、情報処理装置の制御方法、及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4113453A4

Also Published As

Publication number Publication date
JP7635778B2 (ja) 2025-02-26
CN115315730A (zh) 2022-11-08
US20230128305A1 (en) 2023-04-27
EP4113453A4 (en) 2023-08-30
EP4113453A1 (en) 2023-01-04
US12219122B2 (en) 2025-02-04
JPWO2021199735A1 (https=) 2021-10-07

Similar Documents

Publication Publication Date Title
CN103491331B (zh) 显示控制设备、显示控制方法
JP2019160318A (ja) 情報処理装置、情報処理方法、及びプログラム
WO2022107669A1 (ja) 情報処理装置、情報処理方法、情報処理システム
KR101518814B1 (ko) 360도 서라운드 뷰 영상 데이터를 이용한 방송 컨텐츠의 저작방법
JP5701017B2 (ja) 動画再生装置、動画再生方法、コンピュータプログラム、記憶媒体
JP2012060218A5 (https=)
US12212883B2 (en) Information processing devices, methods, and computer-readable medium for performing information processing to output video content using video from mutiple video sources
JP7667483B2 (ja) 情報処理装置、動画合成方法及び動画合成プログラム
JP3951778B2 (ja) 映像コンテンツ編集支援システム，撮像装置，編集者端末装置，記録媒体，プログラム，映像コンテンツ編集支援方法
US10375456B2 (en) Providing highlights of an event recording
WO2009150827A1 (ja) コンテンツ編集装置
JP7655310B2 (ja) 情報処理装置、情報処理方法、プログラム
JP7635778B2 (ja) 情報処理装置、画像処理システム、情報処理方法
US20220286642A1 (en) Event-oriented multi-channel video backup apparatus and method, and network surveillance camera system including the same
JP2021197082A (ja) 情報処理装置、情報処理装置の制御方法及びプログラム
US12470679B2 (en) Information processing device, information processing method, program, and display system
JP7827064B2 (ja) 情報処理装置、情報処理方法、プログラム
KR20100092474A (ko) 편집 장치, 편집 방법, 및 편집 프로그램
GB2536032A (en) Method for processing video clip for live-broadcast replay
JPH1051734A (ja) 動画像編集装置および動画像編集方法
WO2021199715A1 (ja) 情報処理装置、情報処理方法、プログラム
JP2008305495A (ja) コンテンツ送出装置
US10911839B2 (en) Providing smart tags
JP2025162859A (ja) 画像処理システム、及び画像処理方法
HK40037763A (en) Method and device for synthesizing video resource, storage medium and electronic device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21781577

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022511634

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2021781577

Country of ref document: EP

Effective date: 20220926

NENP Non-entry into the national phase

Ref country code: DE