JP2019050463A - Image processing system and method for controlling image processing system - Google Patents

Abstract

To read out, in a short time, a frame photographed at a specific time from a plurality of images photographed by a plurality of imaging means.SOLUTION: An image processing system comprises: a plurality of imaging means (112) that respectively photograph a plurality of images of a subject from a plurality of directions; control information generation means (120) that generates control information added to frames of the plurality of images photographed by the plurality of imaging means on the basis of time information and recording instructions superimposed to the frames of the plurality of images photographed by the plurality of imaging means; writing control means (230) that writes, in storage means, the frames of the plurality of images photographed by the plurality of imaging means on the basis of the control information; read-out control means (270) that reads out the frames of the plurality of images from the storage means on the basis of the control information; and virtual viewpoint image creation means (270) that creates a virtual viewpoint image by using the frames of the plurality of images read out by the read-out control means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing system and a control method of the image processing system.

  Recently, attention has been focused on a technique of setting a plurality of cameras at different positions and synchronously photographing with multiple viewpoints, and generating virtual viewpoint content using a plurality of viewpoint images obtained by photographing. According to the above-mentioned technology, for example, it is possible to view a specific scene (for example, a goal scene) such as soccer or basketball from various angles, thereby providing the user with a high sense of reality as compared with a normal image. it can. Multiple cameras capture images. An image processing unit such as a server aggregates images captured by a plurality of cameras, performs processing such as three-dimensional model generation and rendering, and transmits the processed image to a user terminal. Thereby, generation and browsing of virtual viewpoint content based on a plurality of viewpoint images can be realized.

  At this time, the number of images collected on the server is enormous. The server takes a long time to extract images taken at the same time from a group of images taken by a plurality of cameras. The extraction time affects the time until the virtual viewpoint content becomes viewable. For example, when the number of cameras is 100 and the video frame rate is 60 fps, about 90 minutes of soccer match shooting, approximately 32.4 million images are collected on the server. In addition, when the image analysis unit separates the foreground and the background of a player or the like and separately manages the respective images, the number of images further increases. Under such circumstances, it is important to be able to generate and view virtual viewpoint content in a short time.

  Patent Document 1 discloses a technique of adding a time code to images taken by a plurality of cameras, extracting and joining images of the same time code to generate a panoramic image.

JP 2002-209208 A

  However, in the technique of Patent Document 1, as image search using a time code, the time code comparison is performed on each image one by one, so the processing time becomes very long.

  An object of the present invention is to provide a method capable of reading out a frame captured at a specific time in a short time from among a plurality of images captured by a plurality of imaging means.

  In the image processing system according to the present invention, a plurality of image pickup means for respectively photographing a plurality of images from a plurality of directions with respect to a subject, and time information superimposed on each frame of the plurality of images photographed by the plurality of image pickup means And control information generating means for generating control information to be added to each frame of the plurality of images taken by the plurality of imaging means based on the recording instruction and the plurality of imaging means based on the control information Write control means for writing each frame of the plurality of images into the storage means, read control means for reading the frames of the plurality of images from the storage means based on the control information, and read by the read control means Virtual viewpoint image generation means for generating a virtual viewpoint image using the frames of the plurality of images.

  According to the present invention, a frame captured at a specific time can be read out in a short time from among a plurality of images captured by a plurality of imaging means.

It is a figure showing an example of composition of an image processing system. It is a figure which shows the structural example of a camera adapter. It is a figure which shows the structural example of an image computing server. It is a flowchart which shows the operation example of a time code extraction part. It is a figure which shows the structural example of image additional information. It is a figure for demonstrating the production | generation method of the image additional information based on recording instruction | indication. It is a figure for demonstrating a memory structure parameter. It is a flowchart which shows the operation example of a write-in control part. It is a figure which shows the structural example of SQ table information. It is a flowchart which shows the operation example of a read-out control part.

  FIG. 1 is a diagram showing an exemplary configuration of an image processing system 100 according to an embodiment of the present invention. The image processing system 100 installs a plurality of cameras and microphones in a facility such as a stadium or a concert hall, and performs shooting and sound collection. The image processing system 100 includes a plurality of sensor systems 110 a to 110 z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190.

  The controller 300 has a control station 310 and a virtual camera operation UI 330. The control station 310 performs operation state management, parameter setting control, and the like on the blocks constituting the image processing system 100 through the networks 310 a to 310 c, 180 a, 180 b, and 170 a to 170 y. Here, the network may be IEEE standard compliant GbE (Gigabit Ethernet) or 10 GbE, or may be a combination of an interconnect Infiniband, industrial Ethernet, or the like. Also, the network is not limited to these, and may be another type of network.

  Sensor systems 110a-110z are connected by a daisy chain of networks 170a-170y. The sensor system 110 z transmits 26 sets of images and sounds of the 26 sets of sensor systems 110 a-110 z to the image computing server 200. In the present embodiment, 26 sets of sensor systems 110 a to 110 z are not distinguished and described as a sensor system 110 unless otherwise specified. The sensor system 110a includes a microphone 111a, a camera 112a, a camera platform 113a, an external sensor 114a, and a camera adapter 120a. The sensor systems 110b to 110z have the same configuration as the sensor system 110a. The devices in each sensor system 110 are also not distinguished unless there is a special description. The number of sensor systems 110 is not limited to twenty-six. In addition, for example, various modifications may exist such as the camera 112 and the camera adapter 120 may be configured as an integrated device, or the external sensor 114 may not be present.

  In the present embodiment, the image is described as including a moving image and a still image unless otherwise noted. That is, the image processing system 100 can process both still images and moving images. Further, although an example in which virtual viewpoint content and virtual viewpoint sound are included in virtual viewpoint content provided by the image processing system 100 will be described, the present invention is not limited thereto. For example, the audio may not be included in the virtual viewpoint content. Also, for example, the sound included in the virtual viewpoint content may be the sound collected by the microphone closest to the virtual viewpoint. Further, in the present embodiment, although the description of the voice is partially omitted for simplification of the description, basically both the image and the voice are processed.

  Each of the sensor systems 110a to 110z has one camera 112a to 112z. That is, the image processing system 100 includes a plurality of cameras (imaging units) 112a to 112z for respectively photographing a plurality of images in a plurality of directions with respect to a subject. The plurality of sensor systems 110 are connected by a daisy chain of networks 170a to 170y. The connection configuration may be a star network configuration in which the sensor systems 110 a to 110 z are connected to the switching hub 180 and mutually transmit and receive data via the switching hub 180.

  The microphone 111a collects voice. The camera 112a captures an image. The camera adapter 120a performs processing on the sound collected by the microphone 111a and the image captured by the camera 112a. Then, the camera adapter 120a transmits the processed audio and image to the camera adapter 120b of the sensor system 110b via the daisy chain network 170a. Similarly, the sensor system 110b transmits the collected sound and the captured image to the sensor system 110c together with the image and the sound acquired from the sensor system 110a. By continuing the operation described above, the sensor system 110z transmits the image and the sound acquired by the sensor systems 110a to 110z to the image computing server 200 via the network 180b and the switching hub 180.

  Next, the configuration and operation of the image computing server 200 will be described. The image computing server 200 processes data acquired from the sensor system 110z. The image computing server 200 includes a front end server 230, a database (hereinafter also referred to as DB) 250, a back end server 270, and a time server 290.

  The time server 290 synchronizes the imaging timings of the plurality of cameras 112. As a result, the image processing system 100 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, and therefore, it is possible to suppress the deterioration in quality of the virtual viewpoint image due to a shift in the imaging timing. The present invention is not limited to the case where the time server 290 manages time synchronization of a plurality of cameras 112, and each camera 112 or each camera adapter 120 may independently perform processing for time synchronization.

  The front end server 230 reconstructs a segmented transmission packet based on the image and sound acquired from the sensor system 110 z and converts the data format, and then, according to the camera identifier, data type, and frame number. , Write to the database 250.

  Next, the back end server 270 receives specification of a viewpoint from the virtual camera operation UI 330, reads out the corresponding image and sound data from the database 250 based on the received viewpoint, performs rendering processing, and generates a virtual viewpoint image Do. More specifically, the back-end server 270 generates virtual viewpoint content based on, for example, image data of a foreground area extracted from images captured by the plurality of cameras 112 and a viewpoint specified by a user operation. The method by which the camera adapter 120 extracts the foreground area will be described later.

  The back end server 270 outputs the rendered image and sound to the end user terminal 190. The end user terminal 190 can perform image browsing and audio viewing in accordance with the designation of the viewpoint according to the user's operation.

  The configuration of the image computing server 200 is not limited to this. For example, at least two of the front end server 230, the database 250, and the back end server 270 may be integrally configured. Further, the image computing server 200 may have a plurality of at least one of the front end server 230, the database 250, and the back end server 270. In addition, devices other than the above-described devices may be included at any position in the image computing server 200. Furthermore, the end user terminal 190 or the virtual camera operation UI 330 may have at least a part of the functions of the image computing server 200.

  Further, the configuration of the image processing system 100 is not limited to the above configuration. For example, the virtual camera operation UI 330 may directly acquire an image from the sensor systems 110a to 110z. Also, the virtual camera operation UI 330 may be configured to directly access the database 250. Further, the image processing system 100 is not limited to the physical configuration described above, and may be logically configured.

  The virtual viewpoint image is an image obtained when the subject is photographed from a virtual viewpoint. In other words, the virtual viewpoint image is an image representing the appearance at the designated viewpoint. The virtual viewpoint (virtual viewpoint) may be designated by the user, or may be automatically designated based on the result of image analysis or the like.

  FIG. 2 is a view showing a configuration example of the camera adapter 120 of FIG. The camera adapter 120 includes a camera control unit 121, a microphone control unit 122, a data transmission / reception unit 123, a time synchronization control unit 124, a foreground / background separation unit 125, a three-dimensional model information generation unit 126, a time code extraction unit 127, and image additional information The generation unit 128 is included.

  The camera control unit 121 is connected to the camera 112 and has a function of controlling the camera 112, acquiring a captured image, providing a synchronization signal, setting a time, and the like. For control of the camera 112, for example, setting and reference of shooting parameters (number of pixels, color depth, frame rate, setting of white balance, etc.), status of the camera 112 (during shooting, stopping, synchronizing, error etc.) Acquisition, start and stop of shooting, or focus adjustment. The present invention is not limited to the case where the camera control unit 121 adjusts the focus on the camera 112. If a removable lens is attached to the camera 112, the camera adapter 120 may be connected to the lens to make direct lens adjustments. In addition, the camera adapter 120 may perform lens adjustment such as zooming via the camera 112. The synchronization signal is provided by using the time synchronized externally with the time server 290 by the time synchronization control unit 124 and providing a photographing timing (control clock) to the camera 112. The time setting is performed by providing the time synchronized with the time server 290 by the time synchronization control unit 124 with a time code (time information) conforming to, for example, the format of SMPTE 12M. The time synchronization control unit 124 generates a time code by external synchronization as a time code superimposed on each frame of the image data received from the camera 112. The format of the time code is not limited to SMPTE 12M, and may be another format. In addition, the camera control unit 121 may assign a time code to the image data received from the camera 112 without providing the time code to the camera 112.

  The microphone control unit 122 is connected to the microphone 111, and has a function of controlling the microphone 111, starting and stopping sound collection, and acquiring collected sound data. The control of the microphone 111 is, for example, gain adjustment and state acquisition. Further, the microphone control unit 122, like the camera control unit 121, provides the microphone 111 with a timing and a time code for voice sampling. As clock information for audio sampling timing, time information from the time server 290 is converted to, for example, a 48 KHz word clock and supplied to the microphone 111.

  The data transmission / reception unit 123 performs data communication with the other camera adapters 120, the front end server 230, the time server 290, and the control station 310 via the networks 170a to 170y, 180a, 180b, 291 and 310a.

  The time synchronization control unit 124 has a function of performing processing relating to time synchronization with the time server 290 in compliance with PTP (Precision Time Protocol) of the IEEE 1588 standard. The time synchronization control unit 124 is not limited to PTP, and may perform time synchronization using other similar protocols.

  The foreground / background separator 125 separates the image data captured by the camera 112 into a foreground image and a background image. That is, the foreground / background separation unit 125 included in each camera adapter 120 extracts a predetermined area from the image captured by the corresponding one of the plurality of cameras 112. The predetermined area is, for example, a foreground image obtained as a result of object detection corresponding to a photographed image. By this extraction, the foreground / background separator 125 separates the photographed image into a foreground image and a background image. The object is, for example, a person. However, the object may be a specific person (such as a player, a director, and / or a referee), or may be an object having a predetermined image pattern such as a ball or a goal. Also, a moving body may be detected as an object. The foreground / background separation unit 125 separates and processes a foreground image including important objects such as a person and a background area not including such objects.

  The three-dimensional model information generation unit 126 uses the foreground image separated by the foreground / background separation unit 125 and the foreground image input from the other camera adapter 120, and uses, for example, the principle of a stereo camera to generate image information related to a three-dimensional model. Generate

  The time code extraction unit 127 extracts auxiliary data to be superimposed on the image data captured by the camera 112. The auxiliary data includes a zoom factor, an exposure, camera parameters such as color temperature, and a time code (hereinafter referred to as an external time code). After determining whether the external time code is broken, the time code extraction unit 127 transmits a normal time code to the image additional information generation unit 128. When the external time code is broken, the time code extraction unit 127 transmits the predicted time code constantly generated internally to the image additional information generation unit 128. The predicted time code is calculated using the previous external time code or predicted time code. At this time, it is desirable that the time code extraction unit 127 calculate the drop frame in consideration.

  The image additional information generation unit 128 generates image data (data such as a foreground image, a background image, and a three-dimensional model) or information to be added to audio data (hereinafter referred to as image additional information). The image additional information includes a time code to be superimposed on image data, a recording sequence number, a recording flag, a frame number, a camera adapter number, a data identifier, and the like. In addition, a part of the parameters included in the image additional information is generated based on the recording instruction input from the control station 310. Image additional information added to image data or audio data is transferred to another camera adapter 120 or front end server 230 via the data transmission / reception unit 123 and used as control information.

  FIG. 3 is a view showing a configuration example of the image computing server 200 of FIG. The image computing server 200 has a front end server 230, a database 250, and a back end server 270. The front end server 230 includes a data input control unit 202 and a write control unit 203. The database 250 includes a communication unit 201, an SQ information management unit 204, and a storage unit 205. The back-end server 270 includes a read control unit 206 and a virtual viewpoint image generation unit 207.

  The communication unit 201 is connected to the control station 310 via a communication path, and transfers control instructions and the like received from the control station 310 to at least one or more functional blocks of the image computing server 200.

  The data input control unit 202 is connected to the camera adapter 120 via the network and the switching hub 180. The data input control unit 202 acquires a foreground image, a background image, a three-dimensional model, and audio data from the camera adapter 120. Further, the data input control unit 202 transfers the data sequentially sent to the write control unit 203 in the subsequent stage. At this time, the data input control unit 202 also transfers to the writing control unit 203 the image additional information added to the image data or the audio data.

  The write control unit 203 writes the data received from the data input control unit 202 at the previous stage to the storage unit 205. The write control unit 203 generates a write address based on the parameters and image additional information preset by the control station 310, and writes data in the area of the storage unit 205 indicated by the write address. At this time, the write control unit 203 determines whether to write according to the state of the recording flag included in the image additional information. That is, the write control unit 203 writes the image data to the storage unit 205 when the recording flag is 1 (during recording), and does not write the image data to the storage unit 205 when the recording flag is 0 (during recording stop). Also, the write control unit 203 monitors the image additional information, and detects the time code when the recording start position is detected, the recording sequence number, the camera adapter number, the data identifier, and the recording start information in which the write address is grouped. Transfer to the information management unit 204.

  The SQ information management unit 204 distributes and manages the recording start information received from the write control unit 203 to the SQ table prepared for each recording sequence number. When the SQ information management unit 204 receives the SQ table read request from the read control unit 206, the SQ information management unit 204 reads all the recording start information (SQ table information) from the SQ table corresponding to the recording sequence number received along with the request, Send to 206.

  The storage unit 205 is a storage medium such as a solid state drive (SSD). The storage unit 205 has a configuration in which SSDs are connected in parallel, and can speed up reading and writing of a large amount of data.

  The read control unit 206 reads the foreground image, the background image, the three-dimensional model, and the audio data from the storage unit 205 and transfers the read data to the virtual viewpoint image generation unit 207. The read control unit 206 generates a read address based on the instructions and parameters received from the control station 310 via the communication unit 201 and the SQ table information received from the SQ information management unit 204. Then, the read control unit 206 reads image data or audio data from the area of the storage unit 205 indicated by the read address.

  The virtual viewpoint image generation unit 207 generates a virtual viewpoint image using the foreground image, the background image, the three-dimensional model, and the audio data received from the read control unit 206. At this time, the virtual viewpoint image generation unit 207 generates a virtual viewpoint image based on parameters such as coordinates of the virtual viewpoint and the viewpoint direction received from the control station 310 through the communication unit 201. The virtual viewpoint image generation unit 207 outputs the generated virtual viewpoint image to the end user terminal 190. Hereinafter, a control method of the image processing system 100 will be described.

  FIG. 4 is a flowchart showing an operation example of the time code extraction unit 127 of FIG. The time code extraction unit 127 performs the process of FIG. 4 when the data packet including the time code is extracted from the auxiliary data to be superimposed on the image data. The data packet is a vertical interval time code (VITC) data packet to be superimposed on horizontal serial data (HANC) of serial digital interface (SDI).

  In step S401, the time code extraction unit 127 detects an error detection code included in the data packet. The error detection code is, for example, a parity bit or a check sum. In step S401, the time code extraction unit 127 calculates an error detection code based on the data value included in the received data packet.

  Next, in step S402, the time code extraction unit 127 compares the error detection code detected in step S401 with the calculated error detection code to determine whether the auxiliary data is corrupted. If the error detection codes of the two do not match (Yes in S402), the time code extraction unit 127 proceeds to the processing step S403 using an external time code because an error in the data packet is not detected. If the two error detection codes do not match (No in S402), the time code extraction unit 127 detects an error in the data packet, and advances the process to processing step S404 using a predicted time code.

  In step S403, the time code extraction unit 127 extracts an external time code from the data packet, and determines the external time code as a time code (hereinafter referred to as an output time code) to be output to the image additional information generation unit 128. Thereafter, the time code extraction unit 127 proceeds with the process to step S405.

  In step S404, the time code extraction unit 127 determines the predicted time code constantly generated internally as the output time code, and the process proceeds to step S405. The predicted time code is a value obtained by adding 1 to the number of frames of the immediately preceding output time code. However, the time code extraction unit 127 needs to calculate the predicted time code in consideration of the drop frame. For example, when the frame rate is 59.94 fps, the time code extraction unit 127 advances the unit of minutes so that the predicted time code is skipped from 00: 00: 00: 59 to 00: 00: 01: 04. Need to skip only 4 frames. However, in the time code extraction unit 127, the unit of minutes is 0, 10, 20, 30, 40, 50 so that the predicted time code advances from 00: 00: 09: 59 to 00: 00: 10: 00. There is no need to skip when moving up to any one.

  In step S405, the time code extraction unit 127 outputs the determined output time code to the image additional information generation unit 128.

  FIG. 5 is a diagram showing an example of the structure of the image additional information 500 generated by the image additional information generation unit 128 of FIG. The image additional information 500 includes a time code 501, a recording sequence number 502, a recording flag 503, a frame number 504, a camera adapter number 505, a data identifier 506, and the like.

  The time code 501 is time information composed of HH (hour): MM (minute): SS (second): FF (frame). The recording sequence number 502 is information indicating the number of times of recording, and is an integer of 0 or more. When the recording sequence number 502 is 0, it means that the camera adapter 120 has not received a recording start instruction from the control station 310 even once.

  The recording flag 503 is information indicating a recording state. When the recording flag 503 is 1, it indicates that recording is in progress, and when the recording flag 503 is 0, it indicates that recording is stopped. The frame number 504 is information indicating the number of frames being recorded, and is an integer of 0 or more.

  The camera adapter number 505 is an identifier for identifying the camera adapter 120 or the camera 112, and is an integer of 0 or more. For example, camera adapter numbers 505 are assigned serial numbers starting from 0 for all camera adapters 120.

  The data identifier 506 is information for identifying a foreground image, a background image, three-dimensional data, and audio data. The data identifier 506 is one alphabetic character (foreground image = F, background image = B, three-dimensional model = M, audio data = A). The configuration of the data identifier 506 is not limited to this, and may be another configuration.

  FIG. 6 is a diagram for explaining a method of generating the image additional information 500 based on the recording instruction of the image additional information generation unit 128 of FIG. The image additional information generation unit 128 generates a recording sequence number 502, a recording flag 503, and a frame number 504 included in the image additional information 500 based on the recording instruction and the parameters. The time code 501 uses the time code received from the time code extraction unit 127 as it is regardless of the recording instruction. The image additional information generation unit 128 is control information generation means, and generates image additional information (control information) 500 to be added to each frame of a plurality of images taken by the plurality of cameras 112.

  First, at time T1, the image additional information generation unit 128 receives a recording start instruction and a recording start time (09: 20: 45: 00) from the control station 310. Starting from this, the image additional information generation unit 128 starts monitoring of the time code received from the time code extraction unit 127, and searches for a time code that matches the recording start time. Note that the state of the recording sequence number 502, the recording flag 503, and the frame number 504 is not changed by the recording start instruction.

  Next, at time T2, the image additional information generation unit 128 detects, from among the time codes received from the time code extraction unit 127, a time code that matches the recording start time. When the additional image information generation unit 128 detects a matching time code, it increments the recording sequence number 502, changes the recording flag 503 from 0 (Low) to 1 (High), and initializes the frame number 504 to 0. Do. Thereafter, the image additional information generation unit 128 holds the values of the recording sequence number 502 and the recording flag 503 until the recording stop instruction is received from the control station 310. Further, each time the image additional information generation unit 128 receives a time code from the time code extraction unit 127, the image additional information generation unit 128 increments the frame number 504.

  Next, at time T3, the image additional information generation unit 128 receives a recording stop instruction from the control station 310. When receiving the recording stop instruction, the image additional information generation unit 128 changes the recording flag 503 from 1 (High) to 0 (Low), and holds the values of the recording sequence number 502 and the frame number 504. Thereafter, the image additional information generation unit 128 holds the values of the recording sequence number 502, the recording flag 503, and the frame number 504 until the next recording start instruction is received.

  The image additional information generation unit 128 sets the recording flag 503 of the frames after the frame to which the time code 501 matching the recording start time is added to 1. Then, when the video recording stop instruction is input, the image additional information generation unit 128 sets the video recording flag 503 of the plurality of cameras 112 to which the frame is added to 0.

  FIG. 7 is a diagram for explaining the memory structure parameter 700. The control station 310 transmits the memory structure parameter 700 to the write control unit 203 or the read control unit 206 via the communication unit 201. The memory structure parameter 700 includes BASE_ADDR 701, CAM_OFST_ADDR 702, BG_OFST_ADDR 703, ML_OFST_ADDR 704, and AD_OFST_ADDR 705. Furthermore, the memory structure parameter 700 includes PRIM_FG_OFST_ADDR 706, PRIM_BG_OFST_ADDR 707, PRIM_ML_OFST_ADDR 708, PRIM_AD_OFST_ADDR 709, and the like. The memory structure parameter 700 is used to write image data or audio data to a predetermined area of the storage unit 205.

  BASE_ADDR 701 is the head address of the area secured for recording the image data captured by all the cameras 112 and the audio data collected by all the microphones 111.

  The CAM_OFST_ADDR 702 is a relative address for specifying the start position of each of the storage areas A (720 a to 720 z). The CAM_OFST_ADDR 702 is used to equally divide the entire area of the storage unit 205 by the number of cameras 112 and to specify the head position of each storage area A (720a to 720z). The CAM_OFST_ADDR 702 is also used to specify the start position of the storage area B (730) storing the foreground image.

  BG_OFST_ADDR 703 is a relative address for specifying the start position of storage area C (740) storing the background image. The BG_OFST_ADDR 703 divides each of the storage areas A (720a to 720z) according to the data type of the foreground image, background image, 3D model, and audio data, and stores the background image in the storage area C (740). It is used to identify the start position.

  The ML_OFST_ADDR 704 is a relative address for specifying the start position of the storage area D (750) storing the three-dimensional model. The ML_OFST_ADDR 704 divides each of the storage areas A (720a to 720z) according to the data type of the foreground image, background image, 3D model, and audio data, and stores the 3D model in the storage area D (750). It is used to specify the beginning position of.

  AD_OFST_ADDR 705 is a relative address for specifying the start position of the storage area E (760) storing audio data. The AD_OFST_ADDR 705 divides each of the storage areas A (720a to 720z) according to the data type of the foreground image, background image, three-dimensional model, and audio data, and stores the audio data in the storage area E (760). It is used to identify the start position.

  The PRIM_FG_OFST_ADDR 706 is a relative address for specifying the head position of each of the storage areas F (731a to 731z) storing the foreground image of one frame. PRIM_FG_OFST_ADDR 706 is used to specify the start position of each foreground image frame sequentially written from the start position of storage area B (730).

  PRIM_BG_OFST_ADDR 707 is a relative address for specifying the head position of each of the storage areas G (741a to 741z) storing the background image of one frame. The PRIM_BG_OFST_ADDR 707 is used to specify the start position of each background image frame sequentially written from the start position of the storage area C (740).

  The PRIM_ML_OFST_ADDR 708 is a relative address for specifying the head position of each of the storage areas H (751 a to 751 z) storing the three-dimensional model of one frame. PRIM_ML_OFST_ADDR 708 is used to specify the start position of each three-dimensional model frame sequentially written from the start position of storage area D (750).

  PRIM_AD_OFST_ADDR 709 is a relative address for specifying the head position of each of the storage areas I (761a to 761z) storing audio data of one frame. PRIM_AD_OFST_ADDR 709 is used to specify the start position of each audio data frame sequentially written from the start position of storage area E (760).

  A frame includes data of a plurality of data types, and is, for example, two or more data of a foreground image, a background image, a three-dimensional model, and audio data. The image additional information generation unit 128 adds the image additional information 500 to each of the plurality of data types of each frame.

  FIG. 8 is a flowchart showing an operation example of the write control unit 203 of FIG. The write control unit 203 performs the process of FIG. 8 each time image data or audio data of one frame is received from the data input control unit 202 of the previous stage. However, the present invention is not limited to this example. For example, the process of FIG. 8 may be performed each time image data corresponding to a GOP (Group Of Picture) is input, and one second of image data or audio data is The process of FIG. 8 may be performed each time an input is made.

  In step S801, the writing control unit 203 acquires the image additional information 500 added to the image data or audio data received from the data input control unit 202. Next, in step S802, the write control unit 203 determines whether to write the image data to the storage unit 205 according to the state of the recording flag 503 included in the image additional information 500. When the recording flag 503 is 1 (during recording) (Yes in S802), the writing control unit 203 needs to write the image data and the audio data to the storage unit 205, and thus the processing proceeds to step S803. When the recording flag 503 is 0 (during recording stop) (No in S802), the writing control unit 203 ends the processing because it is not necessary to write the image data and the audio data to the storage unit 205.

  In step S 803, the write control unit 203 generates a write address WRITE_ADDR based on the memory structure parameter 700 and the image additional information 500. That is, when the recording flag 503 added to the input image data is 1, the write control unit 203 generates an address WRITE_ADDR indicating a storage area of the storage unit 205 for storing the image data. The write control unit 203 generates a write address WRITE_ADDR using Expressions (1), (2), and (3).

  First, OFST_ADDR in equation (1) will be described. When the data identifier 506 is F, the write control unit 203 sets 0 as OFST_ADDR. When the data identifier 506 is B, the write control unit 203 sets BG_OFST_ADDR 703 as OFST_ADDR. When the data identifier 506 is M, the write control unit 203 sets ML_OFST_ADDR 704 as OFST_ADDR. When the data identifier 506 is A, the write control unit 203 sets AD_OFST_ADDR 705 as OFST_ADDR.

  Next, PRIM_OFST_ADDR of Formula (2) is demonstrated. When the data identifier 506 is F, the write control unit 203 sets PRIM_FR_OFST_ADDR 706 as PRIM_OFST_ADDR. When the data identifier 506 is B, the write control unit 203 sets PRIM_BG_OFST_ADDR 707 as PRIM_OFST_ADDR. When the data identifier 506 is M, the write control unit 203 sets PRIM_ML_OFST_ADDR 708 as PRIM_OFST_ADDR. When the data identifier 506 is A, the write control unit 203 sets PRIM_AD_OFST_ADDR 709 as PRIM_OFST_ADDR.

  Thereafter, the write control unit 203 generates a write address WRITE_ADDR according to Expression (3).

  Next, in step S804, the write control unit 203 monitors the image additional information 500, and determines whether the image data or audio data of the target frame is the recording start position. When the target frame is the recording start position (Yes in S804), the writing control unit 203 needs to send the recording start information to the SQ information management unit 204, and therefore the process proceeds to step S805. If the target frame is not the recording start position (No in S804), the writing control unit 203 does not need to send the recording start information to the SQ information management unit 204, and advances the process to step S806. When the recording flag 503 is 1 (during recording) and the frame number 504 is 0, the writing control unit 203 determines that the target frame is the recording start position. However, since the determination using the recording flag 503 has already been performed in step S802, the writing control unit 203 may perform the determination using only the frame number 504 in step S804.

  In step S805, the write control unit 203 uses the time code 501 at the time of detection of the recording start position, the recording sequence number 502, the camera adapter number 505, the data identifier 506, and the recording start information including the write address in a lump. Send to 204. The contents of the recording start information that the write control unit 203 transmits to the SQ information management unit 204 differ depending on the memory structure and the like described with reference to FIG. Further, for example, even if the data identifier 506 is not included in the recording start information, an effect that data can be read faster than in the prior art can be obtained.

  Next, in step S806, the write control unit 203 writes a frame of image data or audio data in the area of the storage unit 205 indicated by the write address WRITE_ADDR generated in step S803. The writing control unit 203 writes data (foreground image, background image, three-dimensional model, and audio data) of a plurality of data types of each frame of the plurality of cameras 112.

  FIG. 9 is a diagram showing the structure of SQ table information 900 managed by the SQ table of the SQ information management unit 204 of FIG. The SQ table information 900 includes a camera number 901, a recording start time 902, and a base address 903.

  The camera number 901 is a number assigned to the camera adapter 120. In the camera number 901, a camera adapter number 505 included in the recording start information received from the writing control unit 203 is set.

  The recording start time 902 is time information composed of HH (hour): MM (minute): SS (second): FF (frame). In the recording start time 902, a time code 501 of a recording start frame included in the recording start information received from the writing control unit 203 is set.

  The base address 903 is the address of the leading image of the recording sequence written in the storage unit 205. In the base address 903, the write address of the recording start frame included in the recording start information received from the write control unit 203 is set.

  The SQ table information 900 generates SQ table information 900 of each of a foreground image, a background image, a three-dimensional model, and audio data for one recording sequence number 502. Therefore, there is a number of SQ table information 900 corresponding to the combination of the number of recording sequences and the data type. When the recording start information received from the writing control unit 203 is entered into the SQ table information 900, the SQ information management unit 204 becomes an entry destination SQ based on the recording sequence number 502 and the data identifier 506 included in the recording start information. Table information 900 is selected. The SQ information management unit 204 registers the recording start time 902 and the base address 903 for each camera number 901 (for each camera identifier) based on the recording sequence number 502 and the data identifier 506.

  FIG. 10 is a flowchart showing an operation example of the read control unit 206 in FIG. When the read control unit 206 receives a read instruction from the control station 310 via the communication unit 201, the read control unit 206 performs the process of FIG.

  In step S1001, the read control unit 206 acquires the read parameter and the memory structure parameter 700 received together with the read instruction. The read parameter is a parameter for reading out from the storage unit 205 image data or audio data corresponding to a specific scene (for example, a goal scene of soccer). The readout parameter includes at least one of a recording sequence number including image data or audio data corresponding to a specific scene, a start time and an end time of the specific scene, and the like. That is, the read parameter may be information for specifying a recording sequence number.

  Next, in step S1002, the read control unit 206 acquires the SQ table information 900 from the SQ information management unit 204 based on the recording sequence number specified from the read parameter. The read control unit 206 transmits the recording sequence number to the SQ information management unit 204 to obtain the SQ table information 900 of the foreground image, the background image, the three-dimensional model, and the audio data corresponding to the corresponding recording sequence number. Acquire sequentially.

  Next, in step S1003, the read control unit 206 generates a read base address based on the start time of the specific scene included in the read parameter, the memory structure parameter 700, and the SQ table information 900. The read base address is generated for each storage area of the foreground image, the background image, the three-dimensional model, and the audio data provided for each camera on the storage unit 205. Therefore, the number of read base addresses is generated by multiplying the number of data types (foreground image, background image, three-dimensional model, audio data) by the number of camera adapters 120. For example, when 100 camera adapters 120 are installed, 400 read base addresses are generated. The read control unit 206 calculates the read base address READ_BASE_ADDR according to Expressions (4), (5), and (6). As for the recording start time 902 and the base address 903 in the equations (5) and (6), corresponding data is read out from the SQ table information 900 according to the camera number 901 to be processed.

  First, PRIM_OFST_ADDR of equation (4) will be described. When the data type is a foreground image, the read control unit 206 sets PRIM_FR_OFST_ADDR 706 as PRIM_OFST_ADDR. When the data type is a background image, the read control unit 206 sets PRIM_BG_OFST_ADDR 707 as PRIM_OFST_ADDR. When the data type is a three-dimensional model, the read control unit 206 sets PRIM_ML_OFST_ADDR 708 as PRIM_OFST_ADDR. When the data type is voice data, the read control unit 206 sets PRIM_AD_OFST_ADDR 709 as PRIM_OFST_ADDR.

  Further, the read control unit 206 calculates the difference between the start time of the specific scene and the recording start time 902 as the difference frame number DIFF_FRAME_NUM by the equation (5) in consideration of the drop frame. Thereafter, the read control unit 206 calculates the read base address READ_BASE_ADDR according to Expression (6).

  Next, in step S1004, the read control unit 206 calculates the total number of frames of a specific scene. Specifically, the read control unit 206 subtracts the start time from the end time of the specific scene to calculate the total number of frames of the specific scene. However, the read control unit 206 calculates the total number of frames in consideration of a drop frame.

  Next, in step S1005, the read control unit 206 initializes a variable N to zero. The variable N indicates the frame number being processed.

  Next, in step S1006, the read control unit 206 generates a read offset address READ_OFST_ADDR based on the memory structure parameter 700 and the variable N. The readout offset address is generated for each of the storage areas of the foreground image, the background image, the three-dimensional model, and the audio data provided on the storage unit 205. Therefore, readout offset addresses are generated as many as the number of data types (foreground image, background image, three-dimensional model, audio data). The read control unit 206 calculates the read offset address READ_OFST_ADDR according to equation (7).

  Next, in step S1007, the read control unit 206 generates a read address using the read base address READ_BASE_ADDR of Expression (6) and the read offset address READ_OFST_ADDR of Expression (7). The read address is generated by adding the read offset address READ_OFST_ADDR to each read base address READ_BASE_ADDR generated in step S1003. The read addresses are generated by the number of read base addresses READ_BASE_ADDR. As the read offset address READ_OFST_ADDR, an offset address of the same data type as the read base address READ_BASE_ADDR is selected. The read control unit 206 calculates the read address READ_ADDR according to equation (8).

  Next, in step S1008, the read control unit 206 reads image data or audio data from the area of the storage unit 205 indicated by the read address READ_ADDR. The read control unit 206 uses the read address READ_ADDR to store one frame unit of data from the storage area of the foreground image, the background image, the three-dimensional model, and the audio data provided for each camera 112 on the storage unit 205. read out.

  Next, in step S1009, the read control unit 206 increments the variable N. Next, in step S1010, the read control unit 206 determines whether the variable N is smaller than the total number of frames. If the variable N is smaller than the total number of frames, the read control unit 206 does not read all the frames from the storage unit 205, so the process returns to step S1006 to process the next frame. When the variable N is not smaller than the total number of frames, the read control unit 206 ends the process of FIG. 10 because all the frames are read from the storage unit 205.

  The read control unit 206 reads a frame after the start time of a specific scene of the plurality of cameras 112 from the storage unit 205. The read control unit 206 reads data (foreground image, background image, three-dimensional model, and audio data) of a plurality of data types of each frame of the plurality of cameras 112. The virtual viewpoint image generation unit 207 generates a virtual viewpoint image using the frames of the plurality of cameras 112 read by the read control unit 206.

  As described above, the writing control unit 203 and the reading control unit 206 perform writing control and reading control of the storage unit 205 based on the image additional information 500 generated by the camera adapter 120. Thus, the image computing server 200 extracts images taken at the same time from images taken by a plurality of cameras 112 in a short time, and shortens the time until browsing virtual viewpoint content. Can.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  In addition, the said embodiment only shows the example of embodiment in the case of implementing this invention, and the technical scope of this invention should not be limitedly interpreted by these. That is, the present invention can be implemented in various forms without departing from the technical concept or the main features thereof.

121 camera control unit 122 microphone control unit 123 data transmission / reception unit 124 time synchronization control unit 125 foreground / background separation unit 126 three-dimensional model information generation unit 127 time code extraction unit 128 image additional information generation unit 202 data Input unit, 203 writing control unit, 204 SQ information management unit, 205 storage unit, 206 reading control unit, 207 virtual viewpoint image generating unit

Claims (10)

A plurality of image pickup means for respectively photographing a plurality of images from a plurality of directions with respect to the subject;
Based on time information and a recording instruction superimposed on each frame of the plurality of images photographed by the plurality of imaging units, control information to be added to each frame of the plurality of images photographed by the plurality of imaging units is generated Control information generation means for
Writing control means for writing each frame of a plurality of images taken by the plurality of imaging means into the storage means based on the control information;
Reading control means for reading the frames of the plurality of images from the storage means based on the control information;
And a virtual viewpoint image generation unit configured to generate a virtual viewpoint image using the frames of the plurality of images read by the read control unit. The control information includes a recording state, a frame number, and an identifier of the imaging unit.
The writing control means writes a frame indicating that the recording state is in recording to the address of the storage means according to the frame number and the identifier of the imaging means, and writes the start frame of the recording for each identifier of the imaging means Register the address,
The image processing system according to claim 1, wherein the read control unit reads the frames of the plurality of images from the storage unit based on a write address registered for each identifier of the imaging unit. The control information includes the time information,
3. The image processing system according to claim 2, wherein the control information generation unit sets the recording state of frames subsequent to a frame to which time information coincident with the recording start time is added.   4. The image processing system according to claim 3, wherein the control information generation unit, when inputting a recording stop instruction, sets the recording state to be added to the frames of the plurality of images to a recording stop state. . The writing control means registers the time information added to the start frame of the recording for each identifier of the imaging means,
The read control means is configured to read from the storage means based on an address corresponding to a difference between time information added to a recording start frame registered for each identifier of the imaging means and a start time of a read instruction. The image processing system according to claim 3 or 4, wherein a frame after the start time of the read instruction of a plurality of images is read out. Synchronization control means for generating time information by external synchronization as time information to be superimposed on each frame;
The information processing apparatus further comprises: time information generating means for outputting internally generated time information to the control information generating means when an error of the time information generated by the synchronization control means is detected. The image processing system according to any one of 3 to 5. The frame includes data of a plurality of data types,
The control information includes the data type,
The control information generation unit generates the control information to be added for each of data of a plurality of data types of each frame,
The write control means writes data of a plurality of data types of the frame to an address of the storage means according to the data type,
The read control means may read data of a plurality of data types of the frame from the storage means based on a write address registered for each identifier of the imaging means and an address corresponding to the data type. The image processing system according to any one of claims 2 to 6.   8. The image processing system according to claim 7, wherein the data of the plurality of data types is two or more data of a foreground image, a background image, a three-dimensional model, and audio data. The control information includes a recording sequence number,
The write control means registers a write address of the recording start frame for each identifier of the imaging means based on the recording sequence number,
The read control means reads frames of the plurality of images from the storage means on the basis of a write address registered for each identifier of the imaging means corresponding to the recording sequence number of the read instruction. The image processing system of any one of 2-8. A control method of an image processing system having an image having a plurality of image pickup means for respectively photographing a plurality of images from a plurality of directions with respect to a subject.
Based on time information and a recording instruction superimposed on each frame of the plurality of images photographed by the plurality of imaging units by the control information generation unit, to each frame of the plurality of images photographed by the plurality of imaging units A control information generation step of generating control information to be added;
A writing step of writing each frame of a plurality of images captured by the plurality of imaging units into the storage unit based on the control information by the writing control unit;
Reading out the frames of the plurality of images from the storage means based on the control information by the readout control means;
And a virtual viewpoint image generation step of generating a virtual viewpoint image by using the frames of the plurality of read out images by the virtual viewpoint image generation means.

Images