JP2020092301A - Image processing device, image processing system, image processing method, and program - Google Patents

Abstract

To make it possible to control a data amount of an image by suppressing image quality deterioration of a distant subject even if there are multiple subjects at different distances from a camera.SOLUTION: The image processing device includes: a foreground separation unit for obtaining a subject image related to a subject from an input image; a resolution acquisition unit for acquiring resolution of the subject image; a data compression/decompression unit for separating the subject image into one or more resolution components; and a priority calculation unit for calculating a priority for a separated resolution component. The priority calculation unit calculates the priority according to the resolution of the subject represented by the resolution component. The control related to a data amount is performed according to the calculated priority.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing device, an image processing system, an image processing method, and a program.

A technique for generating virtual viewpoint content using a plurality of viewpoint images obtained by a multi-camera system in which a plurality of cameras (imaging devices) are installed at different positions and synchronously shoots from multiple viewpoints has been attracting attention. According to the technique of generating virtual viewpoint content from a plurality of viewpoint images, for example, a soccer or basketball highlight scene can be viewed from various angles, so that the user has a higher sense of realism than normal images. Can be given.

To generate and browse virtual viewpoint content based on a plurality of viewpoint images, images captured by a plurality of cameras are aggregated in an image processing unit such as a server, and the image processing unit performs processing such as three-dimensional model generation and rendering. It can be realized by transmitting to the user terminal. Patent Document 1 proposes a technique of increasing the frame rate when the distance between the camera and the shooting scene is short, and lowering the image resolution in order to reduce the processing load at that time.

JP, 2007-172035, A

Here, when constructing a large-scale multi-camera system, the data amount of image data generated from each viewpoint may exceed the transmission band. In this case, the image quality of the generated virtual viewpoint content is deteriorated due to the unintended loss of image data. As one method of preventing this, it is conceivable to reduce the resolution of the image as described in Patent Document 1.

However, when shooting a sports scene such as soccer or basketball, a plurality of subjects having different distances from the camera may simultaneously exist within the field of view of the camera. Since the pixel density of the target differs between a subject having a short distance from the camera and a subject having a long distance from the camera, the resolution required to obtain a good image quality also differs. Therefore, the method of controlling the resolution of the entire camera as described in Patent Document 1 impairs the image quality of a subject far from the camera.

The present invention has been made in view of such circumstances, and it is possible to suppress image quality deterioration of a distant subject and control the amount of image data even if a plurality of subjects at different distances from the camera are included. The purpose is to

An image processing apparatus according to the present invention includes an image acquisition unit that obtains a subject image of a subject from an input image, a resolution acquisition unit that obtains the resolution of the subject image, and the subject image as one or more resolution components. And a priority calculation means for calculating a priority for the separated resolution component, wherein the priority calculation means is such that the resolution of the subject represented by the resolution component is smaller than the first resolution. When the resolution is 2, a priority higher than the priority calculated in the first resolution is calculated.

According to the present invention, it is possible to set the priority according to the resolution and control the resolution for each subject, and even if a plurality of subjects with different distances from the camera are included, the image quality degradation of the subject with a long distance Can be suppressed and the amount of image data can be controlled.

It is a figure which shows the structural example of the image processing system in this embodiment. It is a figure which shows the structural example of the camera adapter in 1st Embodiment. It is a figure which shows the structural example of the image processing part in 1st Embodiment. 6 is a flowchart illustrating an example of priority generation processing according to the first embodiment. It is a figure which shows the example of a wavelet transformation coefficient. It is a figure explaining the estimation method of a to-be-photographed object's position. It is a figure which shows the parameter for calculating the resolution of a to-be-photographed object. It is a figure which shows the structural example of the transmission part in 1st Embodiment. 6 is a flowchart showing a processing example in a data compression/decompression unit in the first embodiment. 6 is a flowchart showing a processing example in a message control unit in the first embodiment. 6 is a flowchart showing a processing example in a packet generation unit in the first embodiment. It is a figure which shows the structural example of the transmission part in 2nd Embodiment. 9 is a flowchart showing a processing example in a packet control unit in the second exemplary embodiment. It is a figure which shows the classification of the resolution component of a DCT conversion coefficient. It is a figure which shows the example of the hardware constitutions of the camera adapter in this embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
A first embodiment of the present invention will be described. In the first embodiment described below, control of the code amount (data amount of image data) is independently performed for each camera in the image processing system 100 in which a plurality of cameras installed at different positions synchronize and shoot. I do. The code amount control is performed in frame units by setting the upper limit of the code amount per frame. If the code amount of one frame does not exceed the upper limit, reversible (lossless) data is transmitted, and if all the data exceeds the upper limit, information that has less influence on the image quality is given priority and dropped. Let

A system in which a plurality of cameras and microphones are installed in a facility such as a stadium or a concert hall to perform shooting and sound collection will be described with reference to the system configuration diagram of FIG. 1. FIG. 1 is a block diagram showing a configuration example of an image processing system 100 according to an embodiment of the present invention. The image processing system 100 includes sensor systems 110a to 110z, an image computing server 120, a controller 130, a switching hub 140, and an end user terminal 150.

The controller 130 has a control station 131 and a virtual camera operation UI (user interface) 132. The control station 131 performs management of operating states and parameter setting control for each functional unit (block) included in the image processing system 100 through the networks 161a to 161c, 162a, 162b, and 163a to 163y. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard such as Ethernet (registered trademark), or may be configured by combining an interconnect Infiniband, an industrial Ethernet, and the like. Further, the network is not limited to these and may be another type of network.

The operation of transmitting images and sounds obtained by the 26 sets of sensor systems 110a to 110z from the sensor system 110z to the image computing server 120 will be described. In the image processing system 100 according to this embodiment, the sensor systems 110a to 110z are connected by a daisy chain.

In the present embodiment, unless otherwise specified, 26 sets of sensor systems from the sensor system 110a to the sensor system 110z will be referred to as the sensor system 110 without distinction. The devices in each sensor system 110 are also referred to as the microphone 111, the camera 112, the platform 113, the external sensor 114, and the camera adapter 115, unless otherwise specified. Note that FIG. 1 shows an example having 26 sets of sensor systems, but this is an example, and the number of sensor systems is not limited to this.

In addition, in the present embodiment, the word “image” will be described as including the concept of a moving image and a still image unless otherwise specified. That is, the image processing system 100 according to the present embodiment can process both still images and moving images. Also, an example in which the virtual viewpoint content provided by the image processing system 100 includes a virtual viewpoint image and a virtual viewpoint audio will be described, but the invention is not limited to this. For example, the virtual viewpoint content may not include audio. Further, for example, the sound included in the virtual viewpoint content may be the sound collected by the microphone closest to the virtual viewpoint. Further, in the present embodiment, for simplification of description, description of sound is partially omitted, but basically, it is assumed that both image and sound are processed.

Each of the sensor systems 110a to 110z has one camera 112a to 112z. That is, the image processing system 100 has a plurality of cameras for photographing a subject from a plurality of directions. The plurality of sensor systems 110 are connected to each other by a daisy chain. With this connection configuration, it is possible to reduce the number of connection cables and labor for wiring work in the case of increasing the resolution of captured images to 4K or 8K and increasing the image data capacity accompanying the increase in frame rate. Note that the present invention is not limited to this connection form, and may be, for example, a star type network configuration in which the sensor systems 110a to 110z are connected to the switching hub 140 and data is transmitted and received between the sensor systems 110 via the switching hub 140. ..

Further, FIG. 1 shows a configuration in which all of the sensor systems 110 are cascade-connected to form a daisy chain, but the present invention is not limited to this. For example, the plurality of sensor systems 110 may be divided into some groups, and the sensor systems 110 may be connected in a daisy chain in units of the divided groups. Then, the camera adapter 115, which is the end of the division unit, may be connected to the switching hub to input an image to the image computing server 120. Such a configuration is effective, for example, when the sensor system 110 is provided for each floor in a stadium including a plurality of floors. In this case, it is possible to input to the image computing server 120 for each floor or for each half circumference of the stadium, which simplifies the installation and makes the system easy even in a place where it is difficult to connect all the sensor systems 110 with one daisy chain. Flexibility can be achieved.

Further, the control of the image processing in the image computing server 120 is switched according to whether there is one or two or more camera adapters 115 that are daisy-chain connected and input images to the image computing server 120. .. That is, control is switched depending on whether the sensor system 110 is divided into a plurality of groups. When the number of camera adapters 115 for image input is one, the image of the entire circumference of the stadium is generated while image transmission is performed by the daisy chain connection. Therefore, the timing when the image data of the entire circumference is gathered in the image computing server 120. Are synchronized. That is, unless the sensor system 110 is divided into a plurality of groups, synchronization can be achieved.

However, when there are a plurality of camera adapters 115 that perform image input (the sensor system 110 is divided into a plurality of groups), the delay may be different depending on the lane (route) of the daisy chain. Therefore, in the image computing server 120, it is necessary to perform the image processing in the subsequent stage while checking the aggregation of the image data by the synchronization control that waits until the image data of the entire circumference is gathered and synchronizes.

In the present embodiment, each of the sensor systems 110a to 110z has a microphone 111, a camera 112, a platform 113, an external sensor 114, and a camera adapter 115. The configuration of the sensor system 110 is not limited to this, and may include at least one camera adapter 115 and one camera 112 or one microphone 111. Further, for example, the sensor system 110 may be configured by one camera adapter 115 and a plurality of cameras 112, or may be configured by one camera 112 and a plurality of camera adapters 115. That is, the plurality of cameras 112 and the plurality of camera adapters 115 in the image processing system 100 correspond to each other by N to M (N and M are both integers of 1 or more).

Further, the sensor system 110 may include devices other than the microphone 111, the camera 112, the platform 113, the external sensor 114, and the camera adapter 115. Further, the sensor system 110 may be configured, for example, by integrating the camera 112 and the camera adapter 115. Furthermore, the front-end server 121 may have at least a part of the functions of the camera adapter 115. Note that the sensor systems 110a to 110z are not limited to the same configuration, and a part or all of the sensor system 110 may have a different configuration.

The sound collected by the microphone 111a of the sensor system 110a and the image captured by the camera 112a are subjected to image processing by the camera adapter 115a, and then transmitted to the camera adapter 115b of the sensor system 110b through the network 163a. Similarly, the sensor system 110b transmits the collected sound and the captured image together with the image and sound acquired from the sensor system 110a to the sensor system 110c via the network 163b. By continuing this operation, the images and sounds acquired by the sensor systems 110a to 110z are transmitted from the sensor system 110z to the image computing server 120 via the network 162b, the switching hub 140, and the like.

Although the camera system 112 and the camera adapter 115 are separated in the sensor system 110 in the present embodiment, they may be integrated in the same housing. In that case, the microphone 111 may be built in the integrated camera 112, or may be connected to the outside of the camera 112.

Next, the configuration and operation of the image computing server 120 will be described. The image computing server 120 in this embodiment processes the data acquired from the sensor system 110z. The image computing server 120 includes a front end server 121, a database (DB) 122, a back end server 123, and a time server 124.

The time server 124 has a function of delivering a time and a synchronization signal, and delivers the time and a synchronization signal to the sensor systems 110a to 110z via the switching hub 140. The camera adapters 115a to 115z of the sensor systems 110a to 110z that have received the time and the synchronization signal perform image frame synchronization by externally synchronizing (Genlock) the cameras 112a to 112z based on the time and the synchronization signal. That is, the time server 124 synchronizes the shooting timings of the plurality of cameras 112. As a result, the image processing system 100 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, and thus it is possible to suppress deterioration in the quality of the virtual viewpoint image due to a shift in the capturing timing. In addition, in the present embodiment, the time server 124 manages the time synchronization of the plurality of cameras 112, but the present invention is not limited to this, and even if each camera 112 or each camera adapter 115 independently performs the process for time synchronization. Good.

The front-end server 121 reconstructs the segmented transmission packet from the image and sound acquired from the sensor system 110z to convert the data format, and then stores the data in the database 122 according to the camera identifier, the data type, and the frame number. Write. The back-end server 123 receives the designation of the viewpoint from the virtual camera operation UI 132, reads the corresponding image and audio data from the database 122 based on the received viewpoint, and performs a rendering process or the like to generate a virtual viewpoint image.

The configuration of the image computing server 120 is not limited to this. For example, at least two of the front end server 121, the database 122, and the back end server 123 may be integrally configured. Further, a plurality of at least one of the front end server 121, the database 122, and the back end server 123 may be included. Further, other devices may be included at any position in the image computing server 120. Furthermore, the end user terminal 150 and the virtual camera operation UI 132 may have at least a part of the functions of the image computing server 120.

The rendered image is transmitted from the backend server 123 to the end user terminal 150, and the user operating the end user terminal 150 can perform image viewing and audio viewing according to the designation of the viewpoint. That is, the back-end server 123 generates virtual viewpoint content based on captured images (plurality of viewpoint images) captured by the plurality of cameras 112 and viewpoint information. Specifically, the back-end server 123 uses the virtual viewpoint content based on the image data of the predetermined area extracted from the images captured by the plurality of cameras 112 by the plurality of camera adapters 115 and the viewpoint specified by the user operation. To generate. Then, the backend server 123 provides the generated virtual viewpoint content to the end user terminal 150.

The virtual viewpoint content in the present embodiment is content including a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image showing the appearance at the specified viewpoint. The virtual viewpoint (virtual viewpoint) may be designated by the user, or may be automatically designated based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily designated by the user. Further, the image corresponding to the viewpoint specified by the user from the plurality of candidates and the image corresponding to the viewpoint automatically specified by the device are also included in the virtual viewpoint image.

In the present embodiment, the virtual viewpoint content is described as including audio data (audio data), but it does not have to include audio data. In addition, the back-end server 123 sends the virtual viewpoint image to, for example, H.264. It may be compressed and encoded according to an encoding method such as H.264 or HEVC and then transmitted to the end user terminal 150 using the MPEG-DASH protocol. Further, the virtual viewpoint image may be transmitted to the end user terminal 150 without being compressed. For example, the former that performs compression encoding assumes a smartphone or tablet as the end user terminal 150, and the latter assumes a display capable of displaying non-compressed images. That is, the image format may be switched according to the type of the end user terminal 150. Further, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or another transmission method may be used.

As described above, the image processing system 100 has three functional domains: a video collection domain, a data storage domain, and a video generation domain. The image collection domain includes the sensor systems 110a to 110z, the data storage domain includes the front end server 121, the database 122, and the back end server 123, and the image generation domain includes the virtual camera operation UI 132 and the end user terminal 150.

Note that the present invention is not limited to this configuration, and for example, the virtual camera operation UI 132 can directly acquire an image from the sensor systems 110a to 110z. However, in the present embodiment, a method of arranging a data storage function in the middle is adopted instead of a method of directly acquiring an image from the sensor systems 110a to 110z. Specifically, the front-end server 121 converts image data and audio data generated by the sensor systems 110a to 110z and meta information of these data into a common schema and data type of the database 122. As a result, even if the camera 112 of the sensor systems 110a to 110z changes to a camera of another model, the changed difference can be absorbed by the front-end server 121 and registered in the database 122. This can reduce the possibility that the virtual camera operation UI 132 does not operate properly when the camera 112 is changed to another model camera.

Further, the virtual camera operation UI 132 is configured to be accessed via the backend server 123 without directly accessing the database 122. The back-end server 123 performs common processing related to the image generation processing, and the virtual camera operation UI 132 performs the difference portion of the application related to the operation UI. As a result, in the development of the virtual camera operation UI 132, it is possible to focus on the development of the function request of the UI operating device and the UI for operating the virtual viewpoint image to be generated. The back-end server 123 can also add or delete a common process related to the image generation process in response to a request from the virtual camera operation UI 132. This makes it possible to flexibly respond to the request of the virtual camera operation UI 132.

As described above, in the image processing system 100, the back-end server 123 generates a virtual viewpoint image based on the image data based on the shooting by the plurality of cameras 112 for shooting the subject from the plurality of directions. The image processing system 100 according to the present embodiment is not limited to the physical configuration described above, and may be configured logically. This is the end of the description of the outline of the image processing system 100.

Next, the camera adapter 115 will be described. FIG. 2 is a block diagram showing a configuration example of the camera adapter 115. The camera adapter 115 in this embodiment has a function of controlling the code amount. The camera adapter 115 includes a network adapter 210, a transmission unit 220, an image processing unit 230, and an external device control unit 240.

The network adapter 210 has a data transmission/reception unit 211 and a time control unit 212. The data transmission/reception unit 211 performs data communication with other camera adapters 115, front-end servers 121, time servers 124, and control stations 131 via a network or the like. For example, the data transmission/reception unit 211 transfers the foreground image and the background image separated by the foreground/background separation unit 231 from the image captured by the camera 112 to the next camera adapter 115 predetermined according to the processing of the data routing processing unit 222. Output. Each camera adapter 115 outputs the foreground image and the background image, so that a virtual viewpoint image is generated based on the foreground image and the background image captured from a plurality of viewpoints. In the image processing system 100, the camera adapter 115 that outputs the foreground image but does not output the background image may exist.

The time control unit 212 conforms to the Ordinary Clock of the IEEE 1588 standard, for example, and has a function of storing a time stamp of data transmitted/received to/from the time server 124 and a function of performing time synchronization with the time server 124. The time synchronization with the time server may be realized not only by the IEEE1588 standard but also by another EtherAVB standard or an original protocol. In the present embodiment, a NIC (Network Interface Card) is used as the network adapter 210, but the network adapter 210 is not limited to the NIC and other similar interfaces may be used.

The transmission unit 220 has a function of controlling data transmission to the switching hub 140 and the like via the network adapter 210. The transmission unit 220 includes a data compression/decompression unit 221, a data routing processing unit 222, a time synchronization control unit 223, an image/voice transmission processing unit 224, and a data routing information holding unit 225. The data compression/decompression unit 221 applies a predetermined compression method, a compression rate, and a frame rate to the data transmitted/received via the data transmission/reception unit 211, and compresses the data. It has the function of expanding.

The data routing processing unit 222 uses the information held by the data routing information holding unit 225 to determine the routing destination of the data received by the data transmitting/receiving unit 211 or the data processed by the image processing unit 230. In addition, the data routing processing unit 222 has a function of transmitting data to the determined routing destination. The data routing information holding unit 225 has a function of holding address information for determining a transmission destination of data transmitted/received by the data transmission/reception unit 211.

Here, the routing destination is the camera adapter 115 corresponding to the cameras 112 directed to the same gazing point, since it is high in image frame correlation between the cameras, which is suitable for performing image processing. The order of the camera adapters 115 that output the foreground image and the background image in the relay format in the image processing system 100 is determined according to the determination by the data routing processing unit 222 of the plurality of camera adapters 115.

The time synchronization control unit 223 complies with the PTP (Precision Time Protocol) of the IEEE1588 standard, and has a function of performing processing related to time synchronization with the time server 124. Note that the process related to time synchronization may be performed using not only PTP but another similar protocol.

The image/sound transmission processing unit 224 has a function of creating a message for transferring image data or sound data to another camera adapter 115 or the front end server 121 via the data transmission/reception unit 211. The message includes image data or audio data, and meta information of each data. The meta information includes a time code or sequence number at the time of capturing an image or sampling audio, a data type, an identifier indicating an individual camera 112 or microphone 111, and the like.

The image/sound transmission processing unit 224 also receives a message from the other camera adapter 115 via the data transmission/reception unit 211. Then, the image/sound transmission processing unit 224 restores the data information fragmented to the packet size defined by the transmission protocol into image data or sound data according to the data type included in the message. The image data or audio data to be transmitted may be data-compressed by the data compression/decompression unit 221. If the data is compressed when the data is restored, the data compression/decompression unit 221 performs decompression processing.

The image processing unit 230 has a function of processing image data captured by the camera 112 and image data received from another camera adapter 115. The image processor 230 includes a foreground/background separator 231, a priority generator 232, and a calibration controller 233.

The foreground/background separation unit 231 has a function of separating the image data captured by the camera 112 into a foreground image and a background image. That is, the foreground/background separating unit 231 of each of the plurality of camera adapters 115 extracts a predetermined area from the image captured by the corresponding camera 112 among the plurality of cameras 112. The predetermined area is, for example, a foreground image obtained as a result of object detection on the captured image, and the foreground/background separation unit 231 separates the captured image into the foreground image and the background image by this extraction. The object is, for example, a person. The object may be a specific person (player, manager, and/or referee, etc.), or may be an object for which an image pattern such as a ball or goal is predetermined. Further, a moving body may be detected as the object.

The foreground/background separation unit 231 separates and processes the foreground image including an important object such as a person and the background image that does not include the object, so that a portion corresponding to the object of the virtual viewpoint image generated in the image processing system 100 is processed. The image quality can be improved. Further, by separating the foreground and the background by each of the plurality of camera adapters 115, it is possible to distribute the load on the image processing system 100 having the plurality of cameras 112. The predetermined area is not limited to the foreground image, and may be the background image, for example.

The priority generation unit 232 has a function of generating a priority by using the foreground image separated by the foreground background separation unit 231 and the camera parameter. The camera parameters include internal parameters specific to the camera (focal length, sensor pitch, image center, lens distortion parameter, etc.) and external parameters (rotation matrix, position vector, etc.) representing the position and orientation of the camera with respect to the world coordinate system.

The calibration control unit 233 has a function of acquiring image data required for calibration from the camera 112 via the camera control unit 241 and transmitting the image data to the front-end server 121 that performs arithmetic processing related to calibration. Further, the calibration control unit 233 has a function of performing calibration (dynamic calibration) during image capturing on image data acquired from the camera 112 via the camera control unit 241 according to preset parameters. Have. In this embodiment, the calculation process related to the calibration is performed by the front-end server 121, but the node that performs the calculation process is not limited to the front-end server 121. For example, the arithmetic processing may be performed by another node such as the control station 131 or the camera adapter 115 (including another camera adapter 115).

The external device control unit 240 has a function of controlling a device connected to the camera adapter 115. The external device control unit 240 includes a camera control unit 241, a microphone control unit 242, a platform control unit 243, and a sensor control unit 244.

The camera control unit 241 has a function of connecting to the camera 112 and controlling the camera 112, acquiring a captured image, providing a synchronization signal, setting a time, and the like. Control of the camera 112 includes, for example, setting and reference of shooting parameters (number of pixels, color depth, frame rate, white balance, etc.), status of the camera 112 (shooting, stopping, synchronizing, error, etc.). Acquisition, start and stop of shooting, and focus adjustment. In this embodiment, focus adjustment is performed via the camera 112, but when a detachable lens is attached to the camera 112, the camera adapter 115 may be connected to the lens to directly adjust the lens. .. Further, the camera adapter 115 may perform lens adjustment such as zooming via the camera 112.

The synchronization signal is provided by the time synchronization control unit 223 using the time synchronized with the time server 124 and providing the camera 112 with the shooting timing (control clock). The time setting is performed by providing the time synchronized with the time server 124 by the time synchronization control unit 223 with a time code compliant with the format of SMPTE12M, for example. As a result, the time code provided to the image data received from the camera 112 is added. The format of the time code is not limited to SMPTE12M and may be another format. Further, the camera control unit 241 may add the time code to the image data received from the camera 112, without providing the time code to the camera 112.

The microphone control unit 242 is connected to the microphone 111 and has functions of controlling the microphone 111, starting and stopping sound collection, and acquiring sound data collected. The control of the microphone 111 is, for example, gain adjustment, state acquisition, or the like. Further, like the camera control unit 241, the microphone control unit 242 provides the microphone 111 with the timing and time code for audio sampling. As clock information that is the timing of audio sampling, time information from the time server 124 is converted into a word clock of 48 kHz and supplied to the microphone 111.

The platform controller 243 has a function of connecting to the platform 113 and controlling the platform 113. Control of the pan head 113 includes, for example, pan/tilt control and state acquisition. The sensor control unit 244 has a function of connecting to the external sensor 114 and acquiring sensor information sensed by the external sensor 114. For example, when a gyro sensor is used as the external sensor 114, it is possible to acquire information indicating vibration. Then, using the vibration information acquired by the sensor control unit 244, the image processing unit 230 can generate an image with suppressed vibration prior to the processing in the foreground/background separation unit 231.

The vibration information is used, for example, when the image data of the 8K camera is cut out in a size smaller than the original 8K size in consideration of the vibration information and the position of the image data of the adjacent camera 112 is aligned. It As a result, even if the structure vibration of the building propagates to each camera at different frequencies, the alignment is performed by this function provided in the camera adapter 115. As a result, electronically image-stabilized image data can be generated, and the effect of reducing the processing load of positioning for the number of cameras 112 in the image computing server 120 can be obtained. The sensor of the sensor system 110 is not limited to the external sensor 114 but may be a sensor built in the camera adapter 115, and the same effect can be obtained.

This is the end of the description of the configuration of the camera adapter 115. In the camera adapter 115, the priority generation unit 232, the data compression/decompression unit 221, and the image/sound transmission processing unit 224 are closely related to the code amount control. Hereinafter, the configuration and the flow of processing for generating the priority of the code relating to the code amount control will be described with reference to FIGS. 3 and 4.

FIG. 3 is a block diagram showing a configuration example of the image processing unit 230 in the camera adapter 115. 3, blocks having the same functions as the blocks shown in FIG. 2 are designated by the same reference numerals. The calibration control unit 233 performs color correction processing on the input image to suppress color variations between cameras, and shake correction for stabilizing the position of the image against shake caused by camera vibration. Perform processing (electronic anti-vibration processing), etc.

The foreground/background separation unit 231 will be described. The foreground separation unit 311 performs the separation process of the foreground image by comparing the image data of which the position of the image of the camera 112 has been adjusted with the background image 312. The foreground image obtained here is output as a pair of the image data and the offset value of the foreground image region in the entire image (for example, the position of the upper left pixel of the foreground image). The background updating unit 313 generates a new background image using the background image 312 and the image in which the position of the camera 112 has been adjusted, and updates the background image 312 to the new background image. The background cutout unit 314 controls to cut out a part of the background image 312.

The priority generation unit 232 will be described. In the following description, the foreground image is also referred to as a “subject image” as an image region showing the subject. The camera parameter receiving unit 321 receives camera parameters. The camera parameter is, for example, information obtained by the calibration process, and is transmitted and set from the control station 131 to the target camera adapter 115. The resolution acquisition unit 322 acquires the resolution of the subject image using the offset value of the subject image separated by the foreground separation unit 311 and the camera parameter received via the transmission unit 220, and the separation parameter calculation unit 323 and the priority are acquired. It is output to the degree calculation unit 324. The separation parameter calculation unit 323 calculates the separation parameter used by the data compression/decompression unit 221 from the resolution of the subject in the input subject image, and outputs it to the transmission unit 220 and the priority calculation unit 324. The priority calculation unit 324 calculates a priority for each encoded resolution component from the input subject resolution and separation parameters, and outputs the calculated priority to the transmission unit 220.

FIG. 4A is a flowchart showing an example of priority and separation parameter calculation processing by the priority generation unit 232. In step S401, the camera parameter receiving unit 321 inputs the camera parameter indicating the position/orientation of the camera with respect to the shooting plane (ground). In step S402, the foreground separation unit 311 inputs the offset value of the entire original image of the subject image. In step S403, the resolution acquisition unit 322 acquires the resolution of the subject image. Here, the resolution is defined as the number of pixels per unit area representing the surface of the imaged subject. The resolution depends on the sensor pitch of the image sensor, the focal length, and the distance to the subject. For example, the resolution acquisition unit 322 sets the larger value as the resolution of the subject image as the focal length is larger or the sensor pitch is smaller. Further, for example, the resolution acquisition unit 322 sets the smaller value as the resolution of the subject image as the distance to the subject increases.

In step S404, the separation parameter calculation unit 323 calculates the separation parameter. In this embodiment, the image data is separated into resolution components having different frequencies by wavelet transform, and the separation parameter represents the number of wavelet transforms. The wavelet transform process has a direction, and when separated in the horizontal direction, two images having a horizontal size that is half the original image size and the same vertical size are generated as high-frequency and low-frequency resolution components. .. When the images of the resolution components generated here are further separated in the vertical direction, four resolution components whose vertical and horizontal sizes are half the original image size are generated. In this way, the vertical and horizontal conversions are performed once to form one wavelet conversion.

Of the four resolution components generated by the wavelet transform, a resolution component having a low frequency in both the vertical and horizontal directions is called an LL component, and a resolution component having a high frequency in the vertical direction and a low frequency in the horizontal direction is called an LH component. A resolution component having a low frequency in the vertical direction and a high frequency in the horizontal direction is called an HL component, and a resolution component having a high frequency in both the vertical and horizontal directions is called an HH component. In general, the LL component is a reduced image with half the resolution of the original image, the LH component holds vertical edge information, the HL component holds horizontal edge information, and the HH component holds diagonal edge information. ing. Further, when the resolution components are separated, the LL component is wavelet-transformed again to be hierarchically separated into four resolution components.

FIG. 5 shows an example in which the wavelet transform is applied three times. (0,0) is the HH component in the first conversion, (0,1) is the HL component in the first conversion, and (1,0) is the LH component in the first conversion. In FIG. 5, the first element in parentheses indicates the number of times the horizontal low-frequency component is extracted, and the second element indicates the number of times the vertical low-frequency component is extracted. Therefore, (3, 3) is the LL component obtained as a result of three wavelet transforms.

As described above, a wavelet transform is performed once to generate an image having half the resolution before transformation in the vertical and horizontal directions as an LL component. In the present embodiment, the wavelet transform is executed until the resolution of the subject in the LL component falls below a predetermined required resolution <α>. The required resolution is the minimum resolution required to secure the video quality in the generation of the virtual viewpoint image. When the resolution of the input subject image is <d>, the resolution of the LL component after performing the wavelet transform S times is (d×4 ^−S ). Therefore, the minimum number S of times when the resolution of the subject in the LL component is lower than the required resolution <α> is obtained as follows. That is, in the following example, the number S of times as the separation parameter is determined based on the ratio between the resolution <d> of the subject image and the resolution <α> required for the subject.

In step S405, the priority calculation unit 324 calculates the priority P. The priority is set to a larger value as the resolution of the image information represented by the component becomes smaller, that is, as the frequency becomes lower. When the number of times of extracting the low-frequency components in the vertical and horizontal directions described in FIG. 5 is expressed by (Wx, Wy), the priority P is defined by the following equation.

That is, the priority calculation unit 324 calculates a high priority as the number of times (Wx, Wy) of extracting low-frequency components in the horizontal and vertical directions increases, and increases as the resolution <d> of the subject image decreases. Calculate the priority. In the process illustrated in FIG. 4A, the processing order of step S401 and step S402 is out of order, and the processing order of step S404 and step S405 is out of order.

FIG. 4B is a flowchart showing an example of processing in which the resolution acquisition unit 322 acquires the resolution <d> of the subject image in step S403 shown in FIG. 4A. In step S411, the resolution acquisition unit 322 acquires the required resolution α. This value is preferably set in advance and made common to all cameras in order to unify the priority standard among the cameras, and in this case, it is acquired through the network at the same time as the camera parameters.

In steps S412 to S414, the resolution acquisition unit 322 obtains the distance between the viewpoint and the subject. In the present embodiment, it is assumed that all subjects (for example, people) are on the shooting plane, and the lowest pixel in the subject image corresponds to the point where the subject is grounded on the shooting plane. For example, in the case of a person, the lowest pixel in the person's image corresponds to the point where the foot and the shooting plane are in contact.

The processing of steps S412 to S414 will be described. In step S412, the resolution acquisition unit 322 selects the lowest pixel in the subject image as the representative pixel related to the subject image. If there are a plurality of bottom pixels, the central pixel of the plurality of pixels is selected as the representative pixel. Further, when there are a plurality of lowest pixels, the pixel closest to the center of the subject area may be selected as the representative pixel among the plurality of pixels.

In step S413, the resolution acquisition unit 322 obtains an intersection <X> between the line of sight corresponding to the representative pixel and the shooting plane. Since camera parameters are given in advance and the positional relationship between the shooting plane and the viewpoint and the camera posture are known, it is obvious that the intersection can be calculated. In step S414, the resolution acquisition unit 322 calculates the distance between the viewpoint position <U> and the intersection <X>, and sets this as the distance <L> between the viewpoint and the subject position.

FIG. 6 shows a schematic diagram for explaining the method of estimating the subject position. The representative line-of-sight 604 calculated from the position of the representative pixel of the subject 601 is extended from the viewpoint position <U>605 of the camera 602, and the intersection of this and the shooting plane 603 is <X>. It is assumed that this intersection point <X> is the ground contact point 606 of the subject with respect to the photographing plane 603. In the present embodiment, the subject position in the three-dimensional space estimated from the pixel position of the subject image is used to estimate the distance to the subject. However, another method may be used as long as the distance to the subject can be acquired, for example, a distance camera may be used, or another subject position acquisition sensor may be used.

In step S415, the resolution acquisition unit 322 calculates the resolution <d> of the subject located at the distance <L>. Derivation of the subject resolution will be described with reference to FIG. 7. Let T be the focal length (distance between the lens surface and the sensor surface), L be the distance between the lens surface and the subject surface, S be the vertical size of the sensor, and H be the vertical view angle size of the subject surface. In this case, the magnification M (=H/S) is M=L/T, and further, assuming that the number of pixels in the vertical direction of the sensor is h, the vertical sensor pitch U is U=S/h. Therefore, the resolution <d> of the subject is calculated by d=M/U.

Although a method of calculating the resolution using a simple model has been shown here, the resolution may be obtained from a complicated model in consideration of the thickness and combination of lenses when accuracy is required. In addition, the number of pixels per unit area was calculated from the ratio of the size in the vertical direction to the number of pixels (the number of pixels per unit length). The number may be calculated. Furthermore, although the distance between the viewpoint and the intersection of the representative line of sight with respect to the shooting plane is calculated as the distance to the subject, the distance to the subject may be calculated from the parallax between adjacent viewpoints, or the distance may be calculated using a distance sensor. It may be obtained, or another method may be used as long as the distance can be obtained. Further, the resolution is the number of pixels per unit area of the imaged subject, but may be the number of pixels per unit length of the subject. This is the end of the description of the process of acquiring the resolution.

Next, generation of a data transmission packet in consideration of the priority generated as described above and packet control will be described. FIG. 8 is a block diagram showing a configuration example in the transmission unit 220 related to code amount control. FIG. 8 shows the flow of data when transmitting a subject image, and omits the description of other configurations that do not contribute to the description.

The data compression/decompression unit 221 has an image coding unit 811 and a message generation unit 812. The image encoding unit 811 receives the subject image and the separation parameter from the image processing unit 230, separates the subject image into resolution components according to the separation parameter, and encodes the resolution component. That is, the image encoding unit 811 performs wavelet transform on the subject image for the number of wavelet transforms indicated as the separation parameter, and separates the image into resolution components.

The message generation unit 812 receives the resolution component, its specific information, and the priority, generates a message, and outputs the message to the image/audio transmission processing unit 224. Here, the specific information is information for specifying which component of which subject image the corresponding resolution component is captured by which camera, and is used for decoding image data, for example. The message generator 812 stores the resolution component in the data area of the message, and stores the priority and the identification information of the resolution component in the header area.

The image/sound transmission processing unit 224 includes a message control unit 821, a message holding area 822, and a packet generation unit 823. The message control unit 821 handles the coded data with priority generated by the message generation unit 812 as a message, and manages the message held in the message holding area 822 in consideration of the priority. The message control unit 821 also outputs the message to the packet generation unit 823. The packet generation unit 823 decomposes the input message into predetermined size units to generate packets. The packet generated by the packet generator 823 is output to the network adapter 210.

FIG. 9 is a flowchart showing a processing example of the data compression/decompression unit 221. In step S901, the image encoding unit 811 of the data compression/decompression unit 221 receives the subject image and the separation parameter from the image processing unit 230. In step S902, the image encoding unit 811 of the data compression/decompression unit 221 separates and encodes the subject image into resolution components according to the number of wavelet transforms received as the separation parameter. JPEG2000 is used for encoding here. JPEG2000 can separate image data into resolution components by wavelet transformation. As the encoding method, another method may be used as long as the image data can be separated into resolution components such as JPEG2000.

In step S903, the message generation unit 812 of the data compression/decompression unit 221 receives the priority from the image processing unit 230. In step S904, the message generation unit 812 of the data compression/decompression unit 221 adds a corresponding priority to the resolution component generated in step S902 to generate a message. The message generation unit 812 stores the resolution component in the data area, stores the priority and the identification information of the resolution component in the header area, and generates the message. In step S905, the message generation unit 812 of the data compression/decompression unit 221 outputs the generated message. The process shown in FIG. 9 is executed for all the subject images cut out from the frame.

FIG. 10 is a flowchart showing a processing example of the message control unit 821. In step S1001, the message <m> is input from the data compression/decompression unit 221 to the message control unit 821. In step S1002, the message control unit 821 checks the empty area of the message holding area 822 and determines whether the message <m> can be held in the message holding area 822. When it is determined that there is an area for holding the message <m> in the message holding area 822 (Yes), the message control unit 821 stores the message <m> in the message holding area 822 in step S1003 and executes the processing. finish.

On the other hand, in step S1002, when the message control unit 821 determines that the message holding area 822 does not have a free area for holding the message <m> (No), the process proceeds to step S1004. In step S1004, the message control unit 821 determines whether the message <m> has the highest priority among the messages and the message <m> stored in the message holding area 822. That is, the message control unit 821 compares the highest priority among the messages stored in the message holding area 822 with the priority of the message <m>.

When it is determined in step S1004 that the message <m> is the message with the highest priority (Yes), the message control unit 821 outputs the message <m> to the packet generation unit 823 in step S1005, and the processing is performed. finish. On the other hand, if the message control unit 821 determines in step S1004 that the message <m> is not the highest priority message (No), the process advances to step S1006. That is, when the message control unit 821 determines that the message stored in the message holding area 822 has the highest priority message, the process proceeds to step S1006. In step S1006, the message control unit 821 outputs the message with the highest priority in the message holding area 822 to the packet generation unit 823, and the process returns to step S1002.

In this way, the message control unit 821 plays a role of selecting and outputting a message having a high priority while appropriately storing sequentially input messages in the message holding area 822. If the size of the message holding area 822 is small, a desired selection result cannot be obtained depending on the order of arrival of the messages, so a sufficient size is held.

FIG. 11 is a flowchart showing a processing example of the packet generation unit 823. In step S1101, the message <s> is input from the message controller 821 to the packet generator 823. In step S1102, the packet generation unit 823 determines whether the input message <s> can be inserted into the packet currently being generated. The packet has a predetermined size defined in advance, and the packet generation unit 823 confirms, when the message <s> is input, whether or not the size is exceeded.

When it is determined in step S1102 that the message <s> can be inserted into the packet currently being generated (Yes), the packet generation unit 823 inserts the message <s> into the packet and ends in step S1103. On the other hand, when the packet generation unit 823 determines in step S1102 that the message <s> cannot be inserted into the currently generated packet (No), the process proceeds to step S1104. In step S1104, the packet generation unit 823 completes the generation of the packet currently being generated, writes the highest priority in the packet in the header area of the packet, and outputs the packet to the packet generation unit 823. After that, in step S1105, the packet generation unit 823 generates a new packet.

This is the end of the description of the processing of the message control unit 821 and the packet generation unit 823. The above-described processing shown in FIGS. 10 and 11 is executed for all the subject images acquired from the frame. However, in the present embodiment, the upper limit value MAX of the data amount per frame is set in advance. From the maximum value MAX of the amount of data per frame and the size A per packet, the number R of packets that can be output per frame is calculated by the following formula.

Therefore, when the output of the packet in step S1104 shown in FIG. 11 reaches R times, the processing of the current frame is ended. In addition, when the number of output packets does not reach R even after the processing of all the subject images is completed, the message control unit 821 generates the messages stored in the message holding area 822 in descending order of priority. It is output to the unit 823. This process is repeated until the number of output packets reaches R or all messages are output. Through the above processing, it is possible to preferentially transmit a resolution component having a high priority among resolution components with priorities. The message holding area 822 is initialized in units of frames.

According to the first embodiment, the image data of the foreground area (subject image) is separated into resolution components, and when the data amount exceeds the transmission band, the code data is discarded according to the priority according to the resolution. This makes it possible to prevent unintended loss of image information and stabilize the image quality of the virtual viewpoint content regardless of the position of the subject in the foreground. Therefore, even if a plurality of subjects having different distances from the camera are included, it is possible to suppress the image quality deterioration of the subject having a long distance and to appropriately control the data amount of the image.

(Second embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment, the code amount control is independently performed for each camera. However, when the number of subjects cut out for each camera and the distance are greatly different, the code amount required for quality assurance differs for each camera, so it is desirable to control the code amount in the entire system. For example, the larger the number of cut-out subjects, the larger the required code amount. Therefore, in the second embodiment, the camera adapter 115 monitors the saturation of the transmission band and controls the code amount (data amount) of the entire system, thereby further improving the image quality. In the following, in the image processing system according to the second embodiment, only the points different from the above-described first embodiment will be described.

FIG. 12 is a block diagram showing a configuration example inside the transmission unit 220 in the second embodiment. 12, the blocks corresponding to the blocks shown in FIG. 8 are designated by the same reference numerals, and the duplicated description will be omitted. As shown in FIG. 12, in the transmission unit 220 according to the second embodiment, the image/sound transmission processing unit 224 includes a packet control unit 824, a message control unit 821, a message holding area 822, and a packet generation unit 823. It has a packet holding area 825. With such a configuration, code amount control for reducing image quality deterioration in the entire system is performed in consideration of the priorities of the packet transmitted from the upstream of the daisy chain and the packet generated by itself.

In the second embodiment, the packet generator 823 outputs the generated packet to the packet controller 824. The packet control unit 824 receives packets from its own viewpoint and the network from the packet generation unit 823 and the network adapter 210, discards the packets according to the priority, and outputs the packets to the network adapter 210. The packet holding area 825 is an area for appropriately holding the packet received by the packet control unit 824.

13A and 13B are flowcharts showing a processing example of the packet control unit 824. FIG. 13A shows an example of processing for discarding packets according to the priority of the packet. In step S1301, the packet <p> is input to the packet control unit 824. In step S1302, the packet control unit 824 checks the free area of the packet holding area 825 and determines whether or not the packet <p> can be held in the packet holding area 825. When it is determined that there is an area for holding the packet <p> in the packet holding area 825 (Yes), the packet control unit 824 stores the packet <p> in the packet holding area 825 in step S1303 and executes the processing. finish.

On the other hand, in step S1302, when the packet control unit 824 determines that the packet holding area 825 has no free area for holding the packet <p> (No), the process proceeds to step S1304. In step S1304, the packet control unit 824 determines whether the packet <p> has the lowest priority among the packets and the packet <p> stored in the packet holding area 825. That is, the packet control unit 824 compares the lowest priority of the messages stored in the packet holding area 825 with the priority of the packet <p>.

When it is determined in step S1304 that the packet <p> is the message with the lowest priority (Yes), the packet control unit 824 discards the packet <p> and ends the process in step S1305. On the other hand, if the packet control unit 824 determines in step S1304 that the packet <p> is not the message with the lowest priority (No), the process advances to step S1306. That is, when the packet control unit 824 determines that the message with the lowest priority is included in the packets stored in the packet holding area 825, the process proceeds to step S1306. In step S1306, the packet control unit 824 discards the packet with the lowest priority in the packet holding area 825, and returns to step S1302.

FIG. 13B shows an example of a process of temporarily storing a packet flowing through the network and a packet generated from the data of its own viewpoint in the packet holding area 825 and transmitting the packet to the network according to the priority. In step S1311, the packet control unit 824 confirms whether or not a packet can be output to the network adapter 210, and waits until output becomes possible. When it is determined that the output is possible, the packet control unit 824 outputs the packet with the highest priority in the packet holding area 825 to the network adapter 210 in step S1312.

In the present embodiment, an example is shown in which the packet control unit 824 and the packet holding area 825 have a function of controlling packet discard as a part of the image/voice transmission processing unit 224. However, it is not necessary that all the image/sound transmission processing units 224 have the packet discard control function, and some of the image/sound transmission processing units 224 may have the packet discard control function. The packet discard control function may be a device independent of the image/voice transmission processing unit 224, and may be installed alone on the network. The functions of the packet control unit 824 and the packet holding area 825 do not change in any of the forms described here.

Further, in the present embodiment, the processing of separating the subject image into resolution components is realized by DCT transformation instead of wavelet transformation. First, the subject image is divided into 8×8 blocks, and DCT conversion is performed in block units. The DCT coefficient obtained at this time is also obtained in an 8×8 block, and the lower left side indicates lower frequency information. For example, if the upper left 4×4 block is taken out and inverse transformation is performed, an image having half the resolution in each of the vertical and horizontal directions can be obtained. FIG. 14 shows the resolution component of the DCT coefficient when DCT conversion is performed in 8×8 block units. Coefficients having the same number are one resolution component. When the block size is K×K and the number representing the resolution component is i (i<K), the priority P of the resolution component is calculated as follows.

This is an index showing the ratio of the number of pixels of the resolution represented by the i-th resolution component to the original resolution. In this embodiment, the JPEG algorithm is used to acquire the above-described resolution component. The image coding algorithm of JPEG is roughly divided into three steps of block division, DCT transform, and coding of transform coefficients. Here, the coding of transform coefficients is modified. In JPEG, transform coefficients are quantized in zigzag scan order and Huffman coded. In the present embodiment, the transform coefficient is separated into resolution components as shown in FIG. 14, and the transform coefficient is quantized and coded in resolution component units. The code data obtained here is treated as a resolution component. However, the resolution component acquisition method is not limited to this, and the block division may be of any size, and the transform coefficient coding method may also use other entropy coding algorithms such as arithmetic coding instead of Huffman coding. Good. Further, as the resolution component separating method, the example of the DCT transform is shown in the present embodiment, but any method that can similarly be separated into frequency components may be used, and for example, Hadamard transform may be used.

According to the second embodiment, the same effect as that of the first embodiment can be obtained, and by controlling the code amount in the entire network, the deviation of the required code amount between the cameras can be absorbed, and the entire system can be realized. The quality of the obtained image information can be improved.

(Other embodiments)
The hardware configuration of each device that constitutes this embodiment will be described. In the above-described embodiment, an example has been described in which the camera adapter 115 is equipped with hardware such as FPGA and/or ASIC, and the above-described processing is executed by these hardware. The same applies to various devices in the sensor system 110, the front-end server 121, the database 122, the back-end server 123, and the controller 130. However, at least one of these devices may use, for example, a CPU, GPU, DSP, or the like to execute the processing of the present embodiment by software processing.

FIG. 15 is a block diagram showing a hardware configuration of the camera adapter 115 for realizing the functional configuration shown in FIG. 2 by software processing. Devices such as the front-end server 121, the database 122, the back-end server 123, the control station 131, the virtual camera operation UI 132, and the end user terminal 150 can also have the hardware configuration of FIG. The camera adapter 115 has a CPU 1501, a ROM 1502, a RAM 1503, an auxiliary storage device 1504, a display unit 1505, an operation unit 1506, a communication unit 1507, and a bus 1508.

The CPU 1501 controls the entire camera adapter 115 using computer programs and data stored in the ROM 1502 and the RAM 1503. The ROM 1502 stores programs and parameters that do not need to be changed. The RAM 1503 temporarily stores programs and data supplied from the auxiliary storage device 1504, data externally supplied via the communication unit 1507, and the like. The auxiliary storage device 1504 is composed of, for example, a hard disk drive or the like, and stores content data such as still images and moving images.

The display unit 1505 is composed of, for example, a liquid crystal display or the like, and displays a GUI (Graphical User Interface) for a user to operate the camera adapter 115. The operation unit 1506 is composed of, for example, a keyboard, a mouse, etc., and inputs various instructions to the CPU 1501 in response to an operation by the user. The communication unit 1507 communicates with external devices such as the camera 112 and the front end server 121. For example, when the camera adapter 115 is connected to an external device by wire, a LAN cable or the like is connected to the communication unit 1507. Note that when the camera adapter 115 has a function of wirelessly communicating with an external device, the communication unit 1507 includes an antenna. The bus 1508 connects the respective units of the camera adapter 115 and transmits information.

Note that, for example, a part of the processing of the camera adapter 115 may be performed by the FPGA, and another part of the processing may be realized by software processing using a CPU. Further, although the display unit 1505 and the operation unit 1506 are present inside the camera adapter 115 in this embodiment, the camera adapter 115 may not include at least one of the display unit 1505 and the operation unit 1506. Further, at least one of the display unit 1505 and the operation unit 1506 is present as a separate device outside the camera adapter 115, and the CPU 1501 controls the display unit 1505 and the operation control unit 1506. You may operate as a part.

Further, the above-described embodiment has been described focusing on an example in which the image processing system 100 is installed in a facility such as a stadium or a concert hall. Other examples of facilities include, for example, amusement parks, parks, racetracks, bicycle racetracks, casinos, pools, skating rinks, ski areas, live houses, and the like. Further, the event held at various facilities may be held indoors or outdoors. Further, the facility in the present embodiment also includes a facility which is temporarily (for a limited time) constructed.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that each of the above-described embodiments is merely an example of the embodiment in carrying out the present invention, and the technical scope of the present invention should not be limitedly interpreted by these. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

100: Image Processing System 110: Sensor System 112: Camera 114: External Sensor 115: Camera Adapter 120: Image Computing Server 121: Front End Server 122: Database 123: Back End Server 130: Controller 131: Control Station 132: Virtual Camera Operation UI 150: End user terminal 210: Network adapter 211: Data transmission/reception unit 220: Transmission unit 221: Data compression/decompression unit 224: Image/audio transmission processing unit 230: Image processing unit 231: Foreground/background separation unit 232: Priority Generation unit 233: Calibration control unit 240: External device control unit 311: Foreground separation unit 321: Camera parameter reception unit 322: Resolution acquisition unit 323: Separation parameter calculation unit 324: Priority calculation unit 811: Image coding unit 812: Message generation unit 821: Message control unit 822: Message holding area 823: Packet generation unit 824: Packet control unit 825: Packet holding area

Claims (15)

Image acquisition means for obtaining a subject image relating to the subject from the input image;
Resolution acquisition means for acquiring the resolution of the subject image,
Processing means for separating the subject image into one or more resolution components;
And a priority calculation means for calculating the priority for the separated resolution components,
When the resolution of the subject represented by the resolution component is a second resolution smaller than the first resolution, the priority calculation means calculates a priority higher than the priority calculated in the first resolution. An image processing device characterized by. The image according to claim 1, wherein the priority calculation unit calculates a priority for the resolution component based on a resolution of the subject image and a parameter related to the separation of the subject image in the processing unit. Processing equipment. The image processing apparatus according to claim 1, wherein the priority calculated by the priority calculating unit is higher as the distance to the subject is larger. The priority calculation means calculates a high priority as the resolution of the subject image decreases,
The image processing apparatus according to claim 1, wherein the resolution acquisition unit sets a smaller value to the resolution of the subject image as the distance to the subject increases. A parameter receiving unit for acquiring the focal length and the sensor pitch of the subject image,
The priority calculation means calculates a high priority as the resolution of the subject image decreases,
The image according to any one of claims 1 to 3, wherein the resolution acquisition unit sets a larger value to the resolution of the subject image as the focal length is larger or the sensor pitch is smaller. Processing equipment. It has a parameter calculation means for setting a parameter related to the separation of the subject image in the processing means so that the resolution of the subject image can be separated into a smaller resolution as the resolution of the subject image increases. The image processing apparatus according to claim 1. 7. The image processing apparatus according to claim 6, wherein the parameter calculation unit determines the parameter based on a ratio between a resolution of the subject image and a required resolution for the subject in the image. The image processing apparatus according to claim 1, further comprising an encoding unit that encodes the resolution component. Message generating means for generating a message by adding the priority corresponding to the resolution component,
Packet generating means for storing a plurality of the generated messages and generating a packet,
9. The image processing according to claim 1, further comprising a message control unit that outputs the message generated by the message generation unit to the packet generation unit according to the added priority. apparatus. The packet generation means stores the highest priority among the messages stored in the packet as the priority of the packet in the packet,
10. The image processing apparatus according to claim 9, further comprising a packet control unit that discards the packet having the lowest priority when the input packet cannot be held in the holding area. The image processing apparatus according to claim 1, wherein the resolution acquisition unit sets the number of pixels per unit area of the subject as the resolution of the subject image. The image processing apparatus according to claim 1, wherein the resolution acquisition unit sets the number of pixels per unit length of the subject as the resolution of the subject image. A plurality of image processing devices according to any one of claims 1 to 12,
An image processing system comprising: an image generation device that generates a virtual viewpoint image based on image data acquired from a plurality of image processing devices. An image acquisition step of obtaining a subject image relating to the subject from the input image,
A resolution acquisition step of acquiring the resolution of the subject image,
A processing step of separating the subject image into one or more resolution components;
A priority calculation step of calculating a priority for the separated resolution components,
In the priority calculation step, when the resolution of the subject represented by the resolution component is the second resolution smaller than the first resolution, a priority higher than the priority calculated in the first resolution is calculated. An image processing method characterized by: An image acquisition step of obtaining a subject image relating to the subject from the input image;
A resolution acquisition step of acquiring the resolution of the subject image,
A processing step of separating the subject image into one or more resolution components;
Causing the computer to execute a priority calculation step of calculating a priority for the separated resolution components,
Further, in the priority calculation step, when the resolution of the subject represented by the resolution component is the second resolution smaller than the first resolution, a priority higher than the priority calculated in the first resolution is calculated. A program that causes a computer to perform processing.

Images