JP2023077927A - Data transmission device and control method for the same, data transmission system, and program - Google Patents

Abstract

To reduce the concentration of occurrence of data missing on a particular device in data transmission by multiple devices connected in a daisy chain.SOLUTION: A data transmission device connected to a first external device and a second external device through a daisy chain generates packets from transmission target data and sets a bandwidth selected on the basis of the data volume of the transmission target data, to the generated packets from among a plurality of bandwidths in which the bandwidth of data transmission in the daisy chain is divided into a plurality of different bandwidths. The data transmission device transmits the packets received from the first external device and the above-mentioned generated packets to the second external device in the bandwidth that is set for each packet, by a transmission process that transmits, in parallel, packets at a frequency corresponding to the above-mentioned plurality of bandwidths.SELECTED DRAWING: Figure 2

Description

The present invention relates to a data transmission device and its control method, a data transmission system, and a program.

2. Description of the Related Art In recent years, attention has been paid to a technique in which a plurality of imaging devices (cameras) are installed at different positions to perform synchronous imaging from multiple viewpoints, and a virtual viewpoint video is generated using the multi-viewpoint images obtained by the imaging. In general, virtual viewpoint video is often called free viewpoint video, but is called virtual viewpoint video in this specification. According to the technology for generating a virtual viewpoint video from such multiple viewpoint images, for example, the highlight scene of a soccer or basketball game can be viewed from various angles. can give you a feeling.

The generation and viewing of virtual viewpoint videos based on such multiple viewpoint images is achieved by consolidating the multiple viewpoint images captured by multiple cameras into an image processing unit such as a server, and generating and rendering a 3D model in the image processing unit. It can be realized by performing processing such as, and transmitting it to the user terminal. In order to ensure the image quality of the virtual viewpoint video that is generated and viewed in this way, it is necessary to secure a sufficient band for transmitting the video captured by the plurality of cameras to the server.

In Patent Document 1, an image captured by a camera is separated into, for example, foreground data with many changes such as players and balls, and background data with little change such as goalposts and fields. Techniques for transmitting are disclosed. According to the technique disclosed in Patent Document 1, it is possible to reduce the bandwidth for transmitting images from a plurality of cameras connected in a daisy chain.

JP 2019-008429 A

However, in data transmission by a plurality of devices connected in a daisy chain, as the amount of data to be transmitted increases, data may be lost in a specific device. For example, if the amount of image data transmitted by each camera varies depending on the scene, the amount of image data transmitted from multiple cameras increases in highlight scenes, etc., and may temporarily exceed the bandwidth of the transmission path. . In such a case, transmission of all image data from a plurality of cameras cannot be completed within one frame. When multiple cameras are connected in a daisy chain, the image data of the multiple cameras are transmitted to the server in a fixed order (daisy chain connection order). Therefore, when the amount of image data transmitted from a plurality of cameras exceeds the bandwidth of the transmission line, the image data of the camera whose transmission order is set later is always lost. In this way, the loss of image data concentrates on a specific camera, and image data from the specific camera cannot be obtained, which may lead to deterioration in the image quality of the virtual viewpoint image.

The present invention provides a technique for reducing the concentration of devices with data loss in a specific device in data transmission by a plurality of daisy-chained devices.

A data transmission device according to one aspect of the present invention has the following configuration. i.e.
A data transmission device connected to a first external device and a second external device by a daisy chain,
generating means for generating packets from data to be transmitted;
setting means for setting, to the packet generated by the generation means, a band selected based on the data amount of the data to be transmitted from among a plurality of bands obtained by dividing the data transmission band in the daisy chain into different bands; and,
transmitting means for performing transmission in the plurality of bands in parallel by transmitting packets at a frequency corresponding to the plurality of bands, wherein the packet received from the first external device and the packet generated by the generation means; transmitting means for transmitting the received packets to the second external device in a band set for each packet.

According to the present invention, in data transmission by a plurality of devices connected in a daisy chain, it is possible to reduce the concentration of devices in which data loss occurs in a specific device.

1 is a diagram for explaining the configuration of an image processing system according to a first embodiment; FIG. FIG. 2 is a block diagram showing a functional configuration example of the camera adapter according to the first embodiment; FIG. FIG. 2 is a block diagram showing a hardware configuration example of a camera adapter according to the first embodiment; FIG. FIG. 4 is a diagram showing a format example of a header attached to a packet to be transmitted; 4 is a flowchart showing data transmission processing of the camera adapter according to the first embodiment; 4 is a diagram showing an example of an operation sequence of the data transmission system according to the first embodiment; FIG. FIG. 4 is a diagram showing an example of data transmission timing of the data transmission system according to the first embodiment; 9 is a flowchart showing data transmission processing of the camera adapter according to the second embodiment; FIG. 9 is a diagram showing data transmission timings of the data transmission system according to the second embodiment; 4 is a diagram conceptually showing data transmission by transmission class A and transmission class B; FIG.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

<First embodiment>
FIG. 1 is a diagram showing a configuration example of an image processing system according to the first embodiment. The image processing system 100 generates a virtual viewpoint image based on image data (hereinafter referred to as imaging data) from a plurality of imaging devices installed in facilities such as stadiums and concert halls. The image processing system 100 has cameras 112 a to 112 f as imaging devices, camera adapters 120 a to 120 f, a switching hub 180 , an image computing server 200 , a time server 290 and a controller 300 . In the following description, the cameras 112a to 112f are referred to as cameras 112 when there is no need to distinguish between them. Similarly, the camera adapters 120a to 120f are also referred to as the camera adapter 120 when there is no need to distinguish between them. In addition, unless otherwise specified, the term "image" is assumed to include the concepts of moving images and still images. That is, the image processing system 100 can handle both still images and moving images.

In image processing system 100 , controller 300 , image computing server 200 , time server 290 and camera adapter 120 a are connected to switching hub 180 . Also, the camera adapters 120a to 120f are connected by a daisy chain. Although six cameras are shown in FIG. 1, the number of cameras is not limited to this. Alternatively, a plurality of switching hubs 180 may be cascade-connected, and a plurality of daisy-chain-connected cameras may be connected to each switching hub. Cameras 112a-112f are connected to the camera adapters 120a-120f, respectively. Note that here, regarding the daisy-chained camera adapters 120a-120f, the far side from the image computing server 200 is called the upstream side, and the near side is called the downstream side. In FIG. 6, the camera adapter 120a is positioned most downstream, and the camera adapter 120f is positioned most upstream. A plurality of camera adapters 120 connected by a daisy chain constitute a data transmission system according to this embodiment, and transmit imaging data obtained from the plurality of cameras 112 to the image computing server 200 .

The camera adapter 120 has a function of controlling the connected camera 112, acquiring an image captured by the camera 112, providing a synchronization signal to the camera 112, setting the time, and the like. Control of the camera 112 includes, for example, setting and referencing imaging parameters of the camera 112 (number of pixels, color depth, frame rate, white balance setting, etc.), status (imaging, stopping, synchronizing, error, etc.) acquisition, imaging start/stop, focus adjustment, etc. The synchronization signal provision uses the time synchronized with the time server 290 by the camera adapter 120 to provide the camera 112 with imaging timing (control clock). The time setting provides the time when the camera adapter 120 is synchronized with the time server 290, for example, in a time code conforming to the SMPTE12M format. Thereby, the camera adapter 120 can add a time code to the imaging data received from the camera 112 . Note that the format of the time code is not limited to SMPTE12M, and other formats may be used.

Controller 300 controls the imaging of cameras 112a-112f by sending control signals to daisy-chained camera adapters 120a-120f. The imaging data captured by the cameras 112a-112f is transferred to the Image Computing Server 200 via the daisy-chained camera adapters 120a-120f. In the image processing system 100 shown in FIG. 1, the camera adapter 120f farthest from the switching hub 180, that is, the most upstream camera adapter 120f, starts to transmit image data. The imaging data transmitted from the camera adapter 120f is transferred to the image computing server 200 through the camera adapters 120e to 120a in sequence. The image computing server 200 collects image data (multi-viewpoint images) captured by a plurality of cameras 112, performs processing such as three-dimensional model generation and rendering, and creates virtual viewpoint images observed from arbitrary viewpoints. Generate. The generated virtual viewpoint image is transmitted to a user terminal (not shown) and viewed by the user.

FIG. 2 is a block diagram showing a functional configuration example related to transmission/reception processing of the camera adapter 120 that constitutes the data transmission system of this embodiment. In FIG. 2, the transmission data generation unit 121 divides image data based on the captured image obtained by the camera 112 into a predetermined size, and generates packets (also referred to as transmission data) to which headers are added. In this embodiment, it is assumed that the imaging data transmitted and received by the camera adapter 120 is data including a foreground image. A foreground image is an image obtained by extracting an object area (foreground area) from an image captured by the camera 112 and acquired. The object extracted as the foreground area refers to a dynamic object (moving object) that moves (its absolute position and shape can change) when the images are captured from the same direction in time series. Objects are, for example, people such as players and referees in the field where the game is played, for example, a ball in the case of ball games, and singers, performers, performers, and moderators in concerts and entertainment. Also, the object extracted as the foreground area can be, for example, an article used by a person.

In addition, the camera adapter 120 may be configured to further transmit and receive a background image as imaging data. A background image is an image of an area (background area) different from at least the foreground object. Specifically, the background image is an image obtained by removing the foreground object from the captured image. In addition, the background refers to an object to be imaged that is stationary or continues to be nearly stationary when images are captured from the same direction in time series. Such imaging objects are, for example, stages such as concerts, stadiums where events such as competitions are held, structures such as goals used in ball games, and fields. However, the background is at least a region different from the foreground object, and the object to be imaged may include another object or the like in addition to the object and the background.

Port 122 receives packets from camera adapter 120, an external device adjacent upstream in the daisy chain connection. Port 123 sends the packet to camera adapter 120, which is an external device adjacent downstream in the daisy chain.

The transmission/reception processing unit 130 detects the start of one frame based on the vertical synchronization signal from the camera 112 and controls packet transmission/reception by the camera adapter 120 . In the transmission/reception processing unit 130 , a reception data holding unit 131 temporarily holds packets (also referred to as reception data) received by the port 122 in order to ensure the communication performance of the camera adapter 120 . The reception processing unit 132 analyzes the header information of the packet held by the reception data holding unit 131, determines whether to transfer or discard the packet, and performs reception processing such as making changes. . The transmission processing unit 133 performs transmission processing for transmitting the reception data received by the reception processing unit 132 and the transmission data generated by the transmission data generation unit 121 to the downstream camera adapter 120 . The transmission data holding unit 134 temporarily holds the transmission data processed by the transmission processing unit 133 in order to ensure the communication performance of the camera adapter 120 . The count value of the start timer 135 is initialized to 0 at the start of the frame, and then counts up for each clock. When the count value of the start timer 135 reaches a preset value, the transmission processing unit 133 starts transmitting the transmission data generated by the transmission data generation unit 121 . The drop timer 136 has its count value initialized to 0 at the start of a frame, and then counts up for each clock. When the count value of the drop timer 136 reaches a preset value, the transmission processing unit 133 discards packets whose transmission has not been completed (packets whose transmission has not started).

Transmission processing by the transmission processing unit 133 will be further described. The transmission processing unit 133 divides the data transmission band in the daisy chain into different bands, and transmits packets in parallel for each of the divided bands. A plurality of divided bands is hereinafter referred to as a transmission class. As will be described later, a transmission class selected from a plurality of transmission classes based on the amount of data to be transmitted is set in the packet. In the transmission process of this embodiment, two bands of transmission class A and transmission class B are used. FIG. 10 is a diagram conceptually showing data transmission by transmission class A and transmission class B in this embodiment. The transmission processing unit 133 transmits packets with transmission class A set at a frequency corresponding to the bandwidth assigned to transmission class A, and transmits packets with transmission class B set at a frequency equivalent to the bandwidth assigned to transmission class B. send as often as In this manner, in the transmission processing of the transmission processing unit 133, by transmitting packets at a frequency corresponding to a plurality of bands, the transmission of the plurality of bands is performed independently and in parallel. The transmission processing unit 133 transmits packets in a band corresponding to the transmission class by controlling the throughput of storing packets in the transmission data holding unit 134 . In this embodiment, the band of transmission class A is wider than the band of transmission class B, and in FIG. 10, this state is conceptually indicated by the height of the dashed rectangle indicating the transmission class. However, the magnitude of the difference in height of the illustrated rectangles does not correspond to the magnitude of the difference in bandwidth.

Next, the hardware configuration of the camera adapter 120 will be explained using FIG. Note that the hardware configurations of the image computing server 200, the time server 290, and the controller 300 can also be realized by configurations similar to the configuration of the camera adapter 120 described below. The camera adapter 120 has a CPU 211 , a ROM 212 , a RAM 213 , an auxiliary storage device 214 , a display section 215 , an operation section 216 , a communication I/F 217 and a bus 218 .

The CPU 211 implements each function of the camera adapter 120 shown in FIG. 2 by controlling the overall camera adapter 120 using computer programs and data stored in the ROM 212 and RAM 213 . Note that the camera adapter 120 may have one or a plurality of dedicated hardware different from the CPU 211, and at least a part of the processing by the CPU 211 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, data supplied from the outside via the communication I/F 217, and the like. The auxiliary storage device 214 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display unit 215 is composed of, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) for the user to operate the camera adapter 120, and the like. The operation unit 216 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and inputs various instructions to the CPU 211 in response to user's operations. The CPU 211 operates as a display control unit that controls the display unit 215 and as an operation control unit that controls the operation unit 216 .

A communication I/F 217 is used for communication with an external device of the camera adapter 120 . For example, when the camera adapter 120 is connected to an external device by wire (for example, by daisy chain), a communication cable is connected to the communication I/F 217 . If the camera adapter 120 has a function of wirelessly communicating with an external device, the communication I/F 217 has an antenna. A bus 218 connects each part of the camera adapter 120 and transmits information.

In this embodiment, the display unit 215 and the operation unit 216 exist inside the camera adapter 120, but at least one of the display unit 215 and the operation unit 216 exists outside the camera adapter 120 as a separate device. may

FIG. 4(a) is a diagram showing a header format of a header attached to transmission data according to the first embodiment. In the header format of this embodiment, the shaded portion is modified from the general Internet protocol header (IP header) format. In a first embodiment, packets generated from imaging data from cameras 112a-112f are sent to Image Computing Server 200 via daisy-chained camera adapters 120a-120f. Each packet has an IP header appended with a transmission flag and a transmission class. For example, bit 0 (bit16) of the IP header flag field (bit16 to bit18 in FIG. 4(a)) is unused, and is used as a transmission flag in this embodiment. On (1) is set in the transmission flag of the last packet (last fragment) among the imaging data packets transmitted to the downstream camera adapter 120 in one band, and the transmission flags of the other packets are set to ON (1). Off (0) is set. Also, in the IP header of this embodiment, the fragment offset field is reduced by 1 bit and used as a field indicating the transmission class (bit 19 in FIG. 4A). The same band is assigned to each transmission class in the camera adapters 120a to 120f.

FIG. 4B is a diagram showing another format example of the header added to the packet in the first embodiment. As in the IP header format shown in FIG. 4(a), bit 0 of the flag field is used as a transmission flag. The transmission class is represented by 2 bits using 2 bits of the service type field. Since the transmission class in FIG. 4A is represented by 1 bit, it indicates which of the two types of transmission classes (transmission class A or transmission class B). On the other hand, since the transmission class in FIG. 4(b) is represented by 2 bits, up to 4 types of transmission classes can be represented. The bits of the IP packet used to represent the transmission flag and the transmission class are not limited to the examples shown in FIGS. 4(a) and 4(b). Although five or more transmission classes may be used, three or more bits are required to represent the transmission classes.

Each of the plurality of transmission classes is assigned one of a plurality of bands obtained by dividing a band for data transmission by daisy chain into a plurality of different bands. An example using two types of transmission classes (transmission class A and transmission class B) will be described below. Transmission class A and transmission class B are each assigned a bandwidth obtained by dividing the bandwidth for data transmission by the daisy chain, and the transmission class A bandwidth is wider than the transmission class B bandwidth. The transmission processing unit 133 of the camera adapter 120 determines in which band of the transmission class A or B the imaging data should be transmitted according to the data amount of the imaging data of the connected camera 112 . When the transmission processing unit 133 receives a packet in which the same transmission class as the transmission class determined for the imaging data is set and the transmission flag indicates that the packet is the last packet, the transmission processing unit 133 sets the transmission flag of the packet to 0. and transferred to the camera adapter 120 downstream. Then, the transmission processing unit 133 sets a transmission class to a packet generated from the imaging data to be transmitted, and transmits the packet to the downstream camera adapter 120 . At this time, the transmission flag is set to 1 for the last packet among the packets generated from the imaging data to be transmitted.

FIG. 5 is a flow chart showing data transmission processing of the transmission/reception processing unit 130 according to the first embodiment. The transmission/reception processing unit 130 detects the start of a frame by monitoring the vertical synchronization signal of the camera 112 . When the start of a frame is detected, the transmission/reception processing unit 130 initializes the count values of the start timer 135 and the drop timer 136 to 0, and starts the processing shown in FIG. In S500, the transmission data generation unit 121 acquires image data from the camera 112 and generates packets as transmission data from the acquired image data. The transmission data generator 121 sets (sets to 1) the transmission flag of the last packet of the transmission data, and resets (sets to 0) the transmission flags of the other packets. Note that the start timer 135 and the drop timer 136 start counting clocks after being initialized to zero.

In S501, the transmission processing unit 133 determines a transmission class by comparing the amount of transmission data generated by the transmission data generation unit 121 with a preset threshold value, and sets it to each packet, which is transmission data. It should be noted that the transmission class may be determined based on the amount of captured data acquired from the camera 112 . In S502, the transmission processing unit 133 determines whether or not the drop timer 136 has counted up (whether or not the count value of the drop timer 136 has reached a preset value). The set value of the drop timer 136 is set to a value that the drop timer 136 reaches just before the period of one frame ends. If it is determined that the drop timer 136 has counted up (YES in S502), in S513, after the transmission processing unit 133 completes transmission of the packet in the middle of transmission, it discards subsequent untransmitted packets and resumes frame processing. finish. It should be noted that if there is no data whose transmission has not been completed in S513, this processing ends.

If it is determined that the drop timer 136 has not counted up (NO in S502), the transmission processing unit 133 starts transmitting the transmission data generated in S500 based on the count value of the start timer 135 in S503. Determine whether or not If the count value of the start timer 135 has reached a preset value, it is determined to start transmission, and if not, it is determined not to start transmission. In this embodiment, 0 is set to the start timer 135 of the camera adapter 120f positioned at the most upstream, and values that do not reach within one frame period are set to the start timers 135 of the other camera adapters 120a to 120g. . Therefore, only the transmission/reception processing unit 130 of the camera adapter 120f positioned most upstream determines in S503 to start transmission of transmission data. If it is determined to start transmitting the transmission data (YES in S503), in S509 the transmission processing unit 133 transmits the transmission data (packet) generated by the transmission data generation unit 121 in S500. At this time, the transmission processing unit 133 transmits transmission data in the band of the transmission class determined in S501. Thus, the transmission data generated in S500 is immediately transmitted from the most upstream camera adapter 120f to the downstream camera adapter 120e.

In the camera adapter 120 not located in the most upstream, it is determined in S503 that transmission of transmission data is not started (NO in S503), and the process proceeds to S504. In S<b>504 , the reception processing unit 132 checks whether a packet received from the upstream camera adapter 120 via the port 122 is held in the received data holding unit 131 . If the received packet is not held, it is determined that the packet has not been received (NO in S504), and the process returns to S502. On the other hand, if the received packet is held (YES in S504), the reception processing unit 132 determines that the packet has been received, and proceeds to S505. In S505, the reception processing unit 132 notifies the transmission processing unit 133 of the transmission class obtained from the header of the received packet. It is determined whether or not the data transmission classes are the same.

If it is determined that the transmission class of the received packet is the same as the transmission class set in the transmission data (YES in S505), the transmission processing unit 133 sets the transmission flag of the header of the received packet to 1 in S506. Check whether or not The fact that the transmission flag is 1 means that the transfer of the packet from upstream is completed in the transmission class of the transmission data generated in S500. If the transmission flag of the received packet is 1 (YES in S506), in S507 the transmission processing unit 133 changes the transmission flag of the received packet to 0 and stores it in the transmission data holding unit 134. . The packet held in the transmission data holding unit 134 is transmitted to the downstream camera adapter 120 . At this time, the transmission processing unit 133 controls the throughput for storing the received packet in the transmission data holding unit 134, thereby transmitting the packet downstream in the band set for the transmission class of the packet.

In S508, the transmission processing unit 133 determines whether transmission of the transmission data generated by the transmission data generation unit 121 has not yet started. If it is before the start of transmission (YES in S508), in S509, the transmission processing unit 133 transmits the transmission data (packet) generated by the transmission data generation unit 121 downstream. Specifically, when the transmission processing unit 133 stores the transmission data generated in S500 in the transmission data holding unit 134, the transmission data is transmitted downstream from the transmission data holding unit 134 via the port 123. FIG. The transmission processing unit 133 controls the throughput of storing the transmission data generated in S500 in the transmission data holding unit 134, thereby transmitting the transmission data downstream in the band of the transmission class of the transmission data. If transmission of the transmission data generated in S500 has already started (NO in S508), it is not necessary to start transmission of the transmission data, and the process returns to S504.

On the other hand, if it is determined in S506 that the transmission flag of the received packet is 0 (NO in S506), in S510 the transmission processing unit 133 directly transmits the received packet downstream. That is, the transmission processing unit 133 stores the received packet in the transmission data holding unit 134 and the stored packet is transmitted from the transmission data holding unit 134 through the port 123 to the downstream camera adapter 120 . The bandwidth allocation for transmission of the received packet is as described in S507. After that, the process returns to S504.

If it is determined that the transmission class of the received packet is different from the transmission class of the transmission data generated in S500 (NO in S505), in S511, the transmission processing unit 133 sends the received packet directly to the downstream. Send to That is, the transmission processing unit 133 stores the received packet in the transmission data holding unit 134 , and the stored packet is transmitted downstream from the transmission data holding unit 134 via the port 123 . The bandwidth allocation for transmission of the received packet is as described in S507.

In S512, the transmission processing unit 133 confirms whether or not the imaging data to be transmitted, which is obtained from the connected camera 112, is positioned at the most upstream in the transmission class set in S501. For example, when it is confirmed that a packet of the transmission class set in S501 has not been received even after a certain period of time has passed since the start of the frame, the transmission processing unit 133 determines that the captured data of the connected camera 112 is the maximum. Determined to be upstream. It should be noted that the time required for the data to pass through the number of camera adapters 120 is set as the fixed time used for the determination in S512. Therefore, the longer the camera adapter 120 is positioned downstream, the longer the fixed time is set. Also, the elapsed time from the frame start can be obtained from the counter value of the start timer 135 or the drop timer 136, for example. If it is determined that the imaging data from the camera 112 is upstream (YES in S512), the transmission processing unit 133 starts transmitting the transmission data generated by the transmission data generation unit 121 (S508, S509). Note that if it is not determined that the imaging data of the camera 112 is the most upstream (S512), the process returns to S502.

FIG. 6 is a diagram for explaining the operation sequence of data transmission by the image processing system 100 (data transmission system) shown in FIG. A transmission operation by the camera adapter 120 of the image processing system 100 according to the first embodiment will be described with reference to FIG. The camera adapters 120a to 120f each execute the transmission process described with reference to FIG. In FIG. 6, transmission parameter setting, imaging start, frame #1, and frame #2 are arranged along the vertical axis, but the vertical axis does not indicate the passage of time. For example, in the camera adapter 120e, transmission of imaging data of the camera 112f and transmission of imaging data of the camera 112e are substantially performed in parallel.

In S<b>601 , the controller 300 sets camera parameters for determining imaging conditions and transmission parameters for transmission/reception control in the camera 112 via the camera adapter 120 . Here, the controller 300 performs time synchronization of the camera adapter 120 and the camera 112 . In S<b>602 , the controller 300 transmits a control signal for starting imaging of the camera 112 via the camera adapter 120 . The cameras 112a to 112f take images simultaneously according to control signals.

S603 to S610 are frame #1 imaging and transmission operations. In S603, the cameras 112a to 112f take images. In S604, the cameras 112a-112f output one frame of imaging data to the camera adapters 120a-120f. After that, the cameras 112a to 112f repeat imaging for each frame, and output imaging data to the camera adapters 120a to 120f for each frame. Note that the camera adapters 120a to 120f are set with the same value as the threshold of the imaging data amount for determining the transmission class. Imaging data is classified into transmission class A if its data amount is equal to or greater than the threshold, and classified into transmission class B if its data amount is smaller than the threshold. The band in which transmission class A data is transferred is wider than the band in which transmission class B data is transferred. Assume that the captured data of the cameras 112f, 112b, and 112a are classified into the transmission class A, and the captured data of the cameras 112e, 112d, and 112c are classified into the transmission class B in the frame #1.

In S605, the transmission/reception processing unit 130 of the camera adapter 120f starts transmitting the imaging data of the camera 112f. Note that the set value of the start timer 135 is 0 because the camera 112f is positioned upstream in the transmission path. The transmission data generation unit 121 of the camera adapter 120f generates packets obtained from the imaging data as transmission data. The transmission data generation unit 121 of the camera adapter 120f determines the transmission class according to the data amount of the captured data of the camera 112f (or the generated transmission data). In this example, transmission class A is determined, and the transmission data generator 121 sets the determined transmission class to the header of the generated packet. In addition, the transmission data generation unit 121 of the camera adapter 120f sets the transmission flag of the header of the last packet for transmitting the imaging data to 1 (transmission class A), and sets the transmission flag of the header of the exception packet to 0. (transmission class B). The transmission/reception processing unit 130 of the camera adapter 120f transmits transmission data (packets) in a band corresponding to transmission class A to the downstream camera adapter 120e.

The imaging data of the camera 112 f is transferred to the camera adapters 120 e , 120 d , 120 c , 120 b , 120 a in sequence and finally transferred to the image computing server 200 . In the camera adapters 120e, 120d, and 120c, the transmission data generated by the transmission data generation unit 121 is classified into transmission class B, which is different from transmission class A of the transmission data of the camera 112f. Therefore, in the camera adapters 120e, 120d, and 120c, the transmission data (image data of the camera 112f) from the camera adapter 120f is transferred in the transmission class A band while maintaining the state of the transmission flag of the packet.

Since the imaged data of the camera 112b is classified into transmission class A, the camera adapter 120b waits for the completion of the transfer of the transmission class A packet transferred from upstream, and then starts transmitting the imaged data of the camera 112b. . In the example of FIG. 6, the camera adapter 120b transfers the imaging data of the camera 112b from the camera adapter 120c to the camera adapter 120a in the band assigned to transmission class A. When a packet whose transmission class is A and whose transmission flag is 1 is received from the camera adapter 120c, the camera adapter 120b rewrites the transmission flag of the packet to 0 and transmits the packet to the camera adapter 120a. Then, the camera adapter 120b starts transmitting packets generated from the imaging data of the camera 112b in the transmission class A band. In this way, when the camera adapter 120b finishes transmitting the packets classified as transmission class A from the upstream, it starts transmitting the packets of the imaging data of the camera 112b (S606). The transmission flag of the last packet of the imaging data (transmission data) of the camera 112b is set to 1. The imaging data of the camera 112b is transmitted to the camera adapter 120a in the band assigned to transmission class A, and transferred to the image computing server 200 by the camera adapter 120a.

Camera adapter 120 a forwards packets sent from upstream camera adapter 120 b to image computing server 200 . Since the imaged data of the camera 112a is classified into transmission class A, the camera adapter 120a waits for the completion of transfer of the transmission class A packet transferred from upstream, and then starts transmitting the imaged data of the camera 112a. . In addition, the imaging data of the camera 112a is transmitted in the band assigned to the transmission class A. FIG. When a packet with a transmission class of A and a transmission flag of 1 is received from the camera adapter 120b, the camera adapter 120a rewrites the transmission flag of the packet to 0 and transmits the packet to the downstream image computing server 200. do. After that, the camera adapter 120b starts transmitting packets generated from the imaging data of the camera 112a in the transmission class A band (S607). As described above, from the camera adapter 120 a , after the packets of image data of the cameras 112 f and 112 b are transmitted to the image computing server 200 , the packets of image data of the camera 112 a are transmitted to the image computing server 200 .

On the other hand, since the camera adapter 120e has not received a packet of transmission class B for a certain period of time from the start of frame #1, it is determined that the camera adapter 120e is positioned upstream in transmission class B. Start sending imaging data. The camera adapter 120e transmits packets received from the upstream camera adapter 120f downstream in the band assigned to transmission class A, while transmitting packets of imaging data of the camera 112e downstream in the band assigned to transmission class B. Send. The transmission data generation unit 121 of the camera adapter 120e sets the transmission class B determined based on the amount of data in the header of each packet of the imaging data of the camera 112e. In addition, the transmission data generating unit 121 sets the transmission flag=1 in the header of the last packet among the generated packets, and sets the transmission flag=0 in the headers of the other packets. The imaging data of the camera 112 e is transferred to the camera adapters 120 d, 120 c, 120 b, 120 a in the band assigned to transmission class B, and finally transferred to the image computing server 200 .

The camera adapter 120d transfers the transmission class A packet and the transmission class B packet transmitted from the upstream camera adapter 120e to the downstream camera adapter 120c in respective set bands. The imaging data of the camera 112d is classified into transmission class B. Therefore, the camera adapter 120d directly transmits the packet of transmission class A transferred from the upstream to the downstream, waits for the completion of the transfer of the packet of transmission class B transferred from the upstream, and converts the captured data of the camera 112d. to start sending. When a packet whose transmission class is B and whose transmission flag is 1 is received from the camera adapter 120e, the camera adapter 120d rewrites the transmission flag of the packet to 0 and transfers it downstream, and then the camera 112d. imaged data. A packet of imaging data of the camera 112 d is transferred in the order of the camera adapters 120 c, 120 b, 120 a in the band assigned to transmission class B, and finally transferred to the image computing server 200 .

The camera adapter 120c transfers the transmission class A packet and the transmission class B packet transmitted from the upstream camera adapter 120d to the downstream camera adapter 120b in respective bands. The imaging data of the camera 112c is classified into transmission class B. Therefore, similarly to the camera adapter 120d, the camera adapter 120c directly transmits the transmission class A packet transferred from the upstream to the downstream, and waits for the transfer of the transmission class B packet to be completed before the camera 112d captures the image. Start sending data. That is, when a packet with a transmission class of B and a transmission flag of 1 is received, the camera adapter 120c rewrites the transmission flag of the packet to 0 and transfers the packet downstream. Send a packet of data. The imaging data of the camera 112 c is transferred in the order of the camera adapters 120 b and 120 a in the band assigned to transmission class B, and finally transferred to the image computing server 200 .

In FIG. 6, steps S611 to S618 are imaging and transmission operations in frame #2. In S611, the camera 112 starts imaging. In S<b>612 , the camera 112 outputs one frame of imaging data to the camera adapter 120 . Assume that the imaging data of cameras 112e, 112c, and 112b are classified into transmission class A, and the imaging data of cameras 112f, 112d, and 112a are classified into transmission class B in frame #2.

In S613, the camera adapter 120f starts transmitting the imaging data of the camera 112f. Since the camera 112f is positioned at the most upstream position in the transmission path and the set value of the start timer 135 is 0, the downstream transmission of the imaging data of the camera 112f is immediately started. In frame #2, the imaged data of the camera 112f is classified into transmission class B, so transmission class B is set in the header of each packet. Also, the transmission flag=1 is set in the header of the last packet of the imaging data of the camera 112f. The packets of image data of the camera 112f transmitted from the camera adapter 120f are sequentially transferred by the camera adapters 120e, 120d, 120c, 120b, and 120a in the band assigned to transmission class B. Thus, the imaging data of camera 112 f is finally transferred to image computing server 200 .

The imaging data of the camera 112d is classified into the transmission class B for the camera adapter 120d. Therefore, in the camera adapter 120, the packet of transmission class A transferred from the upstream is transmitted to the downstream as it is. is sent downstream. When a packet whose transmission class is B and whose transmission flag is 1 is received from the camera adapter 120e, the camera adapter 120d rewrites the transmission flag of the packet to 0 and transmits it downstream, and then the camera 112d. image data packets. The packets of imaging data of camera 112 d are sequentially transferred by camera adapters 120 c , 120 b , and 120 a in the band assigned to transmission class B, and finally transferred to image computing server 200 .

The camera adapter 120a also operates in the same manner as the camera adapter 120d because the imaging data of the camera 112a is classified as transmission class B. That is, the camera adapter 120a directly transmits the packet of transmission class A transferred from the upstream to the downstream, waits for the completion of the transfer of the packet of transmission class B transferred from the upstream, and converts the imaging data of the camera 112a. to send.

On the other hand, the imaging data of the cameras 112e, 112c, and 112b are classified into transmission class A. Camera adapters 120e, 120c, and 120b transfer packets classified as transmission class B (packets of imaging data of camera 112f and/or camera 112d) transferred from upstream to downstream as they are in the transmission class B band. Forward.

When the camera adapter 120e determines that it is positioned upstream in the transmission class A, it starts transmitting the imaging data of the camera 112e in the transmission class A band. The imaging data packets of the camera 112 e are transferred in the order of the camera adapters 120 d , 120 c , 120 b , 120 a in the band assigned to transmission class A, and finally transferred to the image computing server 200 .

In the camera adapter 120c, the imaging data of the camera 112c is classified into transmission class A. Therefore, the camera adapter 120c directly transmits the packet of transmission class B transferred from the upstream to the downstream, waits for the completion of the transfer of the packet of transmission class A transferred from the upstream, and converts the captured data of the camera 112c. to send. The imaging data of the camera 112 c is transferred in the order of the camera adapters 120 b and 120 a in the band assigned to transmission class A, and finally transferred to the image computing server 200 . In the camera adapter 120b as well, since the imaged data of the camera 112b is classified into the transmission class A, transmission processing similar to that of the camera adapter 120c is performed. That is, the camera adapter 120b directly transmits the packet of transmission class B transferred from the upstream to the downstream, waits for the completion of the transfer of the packet of transmission class A transferred from the upstream, and converts the captured data of the camera 112b. to send. The imaging data of the camera 112b is transferred from the camera adapter 120a to the image computing server 200 in the band assigned to transmission class A.

FIG. 7 shows a timing chart of operations according to the frame #1 and frame #2 transmission operations of the image processing system 100 described above with reference to FIG. However, FIG. 7 shows a case where the drop timer 136 times out. In FIG. 7, the transmission operation of the imaging data of the cameras 112a to 120f by the camera adapters 120a to 120f according to the transmission classes A and B is shown on the time axis. In FIG. 7, the difference in the height of the rectangles of transmission class A and transmission class B conceptually indicates the difference in bandwidth. In other words, transmission class A has a larger bandwidth than transmission class B. However, the magnitude of the difference in height of the illustrated rectangles does not correspond to the magnitude of the difference in bandwidth. In addition, timing deviations in transmission and reception of image data between the camera adapters, which occur when the image data are sequentially transmitted from the camera adapter 120f to the camera adapter 120a, are omitted in FIG. In addition, FIG. 7 omits a shift in the start of transmission of imaging data that occurs in order to confirm that it is the most upstream in one transmission class, a shift in timing that occurs with other processes, and the like.

In frame #1, after the start of the frame, the camera adapter 120f starts transmitting the imaging data of the camera 112f classified into transmission class A. As described in S512, when the camera adapter 120e confirms that it is the most upstream of the camera adapters that transmit imaged data of transmission class B, the camera adapter 120e transmits imaged data of the camera 112e classified into transmission class B. Start. In this way, transmission of imaging data by transmission class A and transmission class B is performed substantially in parallel. Further, in the camera adapter 120d, when the transfer of the imaging data of the camera 112e classified into transmission class B ends at time Td1, the downstream transmission of the imaging data of the camera 112d is immediately started. The imaging data of the camera 112d is transmitted downstream in the band assigned to the transmission class B. FIG.

In the camera adapter 120c, the transfer of the imaging data of the cameras 112e and 112d classified into transmission class B ends at time Tc1, and the transmission of the imaging data of the camera 112c in the band assigned to the transmission class B starts. However, since the drop timer 136 reaches the set value TE1, which is the time immediately before the end of the frame, before the transmission of the imaging data of the camera 112c is completed, all packets of the imaging data of the camera 112c cannot be transmitted. After that, part of the imaging data of the camera 112c is transferred in the order of the camera adapter 120b and the camera adapter 120a. Since the transfer of the imaged data of the camera 112c has been terminated, there are no packets with the transfer flag set to 1 among the packets transferred in the transfer class B. The image computing server 200 analyzes the transferred packets, discards the packets of the imaging data of the camera 112c, and generates a virtual viewpoint image using only the imaging data that has been successfully transferred.

On the other hand, in the camera adapter 120b, when the transfer of the imaged data of the camera 112f classified into the transmission class A ends at time Tb1, the camera adapter 120b transmits the imaged data of the camera 112b in the band assigned to the transmission class A. do. In the camera adapter 120a, when the transfer of the imaged data (imaged data of the cameras 112f and 112b) classified into the transmission class A ends at time Ta1, the camera adapter 120a uses the band assigned to the transmission class A to capture the image of the camera 112a. Send data. The transmission of the imaging data of the camera 112a is completed at time Ta1E, which is earlier than the set value TE1 for dropout.

As described above, in frame #1, only the imaging data of the camera 112c positioned at the most downstream of transmission class B is not completely transmitted within the frame, and part of it is discarded. On the other hand, the imaging data of the cameras 112f, 112b, and 112a classified as transmission class A are all transmitted to the image computing server 200. FIG.

Next, in frame #2, the camera adapter 120f starts transmitting the imaging data of the camera 112f classified as transmission class B. As shown in FIG. In addition, the camera adapter 120e starts transmitting imaged data of the camera 112e classified as transmission class A. In the camera adapter 120d, the transfer of the imaging data classified into transmission class B (the imaging data of the camera 112f) ends at time Td2. At time Td2, the camera adapter 120d starts transmitting the imaged data of the camera 112d in the band assigned to transmission class B. In the camera adapter 120c, the transfer of the imaging data classified into the transmission class A (the imaging data of the camera 112e) ends at time Tc2. The camera adapter 120c starts transmitting the imaging data of the camera 112c in the transmission class A band at time Tc2. Similarly, in the camera adapter 120b, when the transfer of the imaged data (imaged data of the cameras 112e and 112c) classified into the transmission class A ends at time Tb2, the imaged data of the camera 112b is transferred in the band assigned to the transmission class A. transmission is started. The transmission of the imaging data of the camera 112b is completed at time Tb2E, which is earlier than the set value TE2 for dropout.

In the camera adapter 120a, the transfer of the imaging data classified into the transmission class B (the imaging data of the cameras 112f and 112d) ends at time Ta2. The camera adapter 120a starts transmitting the imaging data of the camera 112a in the band assigned to the transmission class B. However, the transmission is not completed during the frame #2 period, and the drop timer 136 reaches TE2, which is set just before the end of the frame. The imaged data of the camera 112a is handled in the same manner as the imaged data of the camera 112c in frame #1.

As described above, in frame #2, the imaging data of the camera 112a positioned at the most downstream of transmission class B was not completely transmitted within the frame and was discarded. On the other hand, all the imaging data of the camera 112 classified into transmission class A could be transmitted to the image computing server 200 .

As described in detail above, according to the first embodiment, imaged data (or transmission data) is classified into transmission class A or B by comparing the amount of data with a preset threshold value, and an appropriate It is transmitted by allocating a band. This ensures that a predetermined amount of imaging data with a large amount of data is transmitted to the image computing server 200 . For example, in frames #1 and #2, although the camera 112 with a large amount of data is different, it is possible to avoid discarding these imaging data. In the present embodiment, there is a high possibility that captured data with a large amount of data is important data for generating a virtual viewpoint image. This is because the captured data may include a foreground image with a large amount of data, that is, there is a high possibility that many important objects corresponding to foreground objects are included. Therefore, according to the present embodiment, it is possible to avoid lack of imaging data that is important for generating a virtual viewpoint image.

In the above description, when the drop timer 136 times out, transmission of a packet in the middle of transmission is completed. However, the present invention is not limited to this. Even if there is a packet in the middle of transmission, the transmission may be stopped immediately. In this case, for example, the image computing server 200 detects a packet whose transmission has been interrupted, and does not use imaging data including that packet for generating a virtual viewpoint image. Also, although the transmission processing unit 133 determines whether or not to discard data by counting up the drop timer 136, the present invention is not limited to this. For example, the transmission processing unit 133 determines whether the transmission can be completed within the frame based on the remaining time and the amount of data in the frame before starting transmission of the imaging data of the camera 112. If it is determined that the transmission cannot be completed, Then, the imaging data may be discarded. As a result, no packets of imaging data that would be discarded are transmitted, so if there is transmission data of another camera adapter that can complete transmission within the remaining time, it can be transmitted. Therefore, the transmission band can be used more effectively. This configuration will be described in a second embodiment.

In the above configuration, packets are always transferred in a band corresponding to the transmission class. Therefore, for example, when all data is classified into transmission class B, the band of transmission class A is not used at all, and a large amount of image data is discarded. Therefore, when the number of image data classified into transmission class A increases, the transmission control of the above embodiment (first mode) is performed, and in other cases, normal transmission control (second mode) is performed. can be switched to do In the first mode, a plurality of bands are transferred in parallel for each band as described above. , are transferred in their original order. For example, in the second mode, the camera adapter 120 sets the transmission class, but does not perform the transmission control for each band of the above embodiment. The image computing server 200 compares the number of imaging data classified into transmission class A and the threshold for each frame. Alternatively, the number of packets classified into transmission class A and a threshold may be compared. The image computing server 200 instructs the controller 300 to execute transmission control in the first mode from the next frame when the number of image data classified into the transmission class A exceeds the threshold. . The controller 300 instructs the camera adapter 120 to perform transmission control in the first mode at the timing of starting the next frame, and the camera adapter 120 performs the transmission control of the above embodiment. Also, when the number of image data classified into transmission class A falls below the threshold, the image computing server 200 instructs the controller 300 to perform transmission control in the second mode from the next frame. . The controller 300 instructs the camera adapter 120 to perform transmission control in the second mode at the timing of starting the next frame, thereby switching the data transmission system to normal transmission control.

<Second embodiment>
In the second embodiment, a configuration will be described in which transmission classes with different frame frequencies are provided and transmission can be multiplexed. The configurations of the image processing system, data transmission system, camera adapter, etc. are the same as those of the first embodiment (FIGS. 1 to 3). In the second embodiment, transmission classes A to C are used as transmission classes for a plurality of bands. Transmission class A and transmission class B are the same as in the first embodiment. The transmission class C band is a predetermined band for transferring transmission data with a frame frequency different from the transmission class A and B frame frequencies. In the transmission class C of this embodiment, captured data is transferred in a predetermined band during a period twice as long as the frame period of other cameras.

In the second embodiment, since there are three transmission classes, the IP header format shown in FIG. 4(b) is used. Of the two bits indicating the transmission class, bit 0 (14 bits) is the same as the transmission class of the first embodiment and indicates whether the transmission class is A or B. FIG. If bit 1 (15 bits) of the transmission class is 1, it indicates that the transmission class is C regardless of the value of bit 0. If bit 1 of the transmission class is 0, either transmission class A or transmission class B is assigned according to the value of bit 0, as in the first embodiment. Transmission of transmission data whose transmission class bit 1 is 1, that is, transmission data of transmission class C is started immediately after the start of the frame. Bands are assigned in advance to each of the transmission classes A to C, and the camera adapters 120a to 120f transmit and receive transmission data using these bands.

FIG. 8 is a flowchart showing data transmission processing by the transmission/reception processing unit 130 in the second embodiment. The same step numbers are attached to the same processing steps as in the first embodiment (FIG. 5). Upon detecting the start of the frame, the transmission/reception processing unit 130 initializes the start timer 135 and the drop timer 136 to 0, and starts counting clocks. Then, the processing shown in FIG. 8 is started.

In S801, the transmission processing unit 133 determines one of transmission classes A, B, and C as the transmission class of the imaging data. Transmission class C is set in the camera adapter 120 in advance, and is not dynamically set according to the amount of data to be transmitted. If the camera adapter 120 is not set to transmission class C, the transmission class is determined in the same manner as in the first embodiment. That is, the transmission processing unit 133 compares the data amount of the transmission data (imaging data) generated by the transmission data generation unit 121 with a preset threshold to determine whether the transmission data is of transmission class A or transmission class B. Decide what is classified. Next, in S802, the transmission processing unit 133 determines whether the transmission class set in S801 is the transmission class C or not. If it is determined that the transmission class is C (YES in S802), the transmission processing unit 133 determines in S508 whether or not transmission is yet to start. If there is before the start of transmission (YES in S508), in S509 the transmission processing unit 133 starts transmitting the imaging data in the band assigned in advance to the transmission class C. FIG.

If it is determined in S504 that a packet has been received (YES in S504), the transmission processing unit 133 determines whether the transmission class of the received packet is C or not in S803. If the received packet is transmission class C (YES in S803), the transmission processing unit 133 stores the received packet in the transmission data holding unit 134 as it is in S510. Thus, the transmission class C packet received from the upstream camera adapter is transmitted to the downstream camera adapter 120 in the band assigned to transmission class C. FIG. In S804, in addition to the processing in S508 of the first embodiment, the transmission processing unit 133 determines whether the transmission of the transmission data generated in S500 will be in time, that is, whether the imaged data of the camera 112 is determined based on the remaining time in the frame and the amount of data. can be completed within the frame. If it is determined that it will be in time (YES in S804), transmission of transmission data is started in S509. If it is determined that it will not be in time (NO in S804), the transmission of the transmission data will not be started, and the transmission data will be discarded after all (S513). Furthermore, in S805, the transmission processing unit 133 executes the above-described processing of S513 for the imaging data of the transmission classes A and B. On the other hand, for the imaging data of transmission class C, different processing is performed for odd-numbered frames and even-numbered frames. That is, the transmission processing unit 133 does not discard the imaging data of transmission class C in odd-numbered frames, and discards the untransmitted imaging data of transmission class C in even-numbered frames. As a result, transmission of imaging data of transmission class C across two frames is realized. In this embodiment, transfer over three frames is shown, but transfer over three or more frames may be performed.

FIG. 9 is a timing chart showing transmission operations in frames #1 and #2 of the second embodiment. In FIG. 9, transmission of imaging data of the cameras 112a to 112f by the camera adapters 120a to 120f is shown according to transmission classes A, B, and C on the time axis. The difference in the height of the rectangles of transmission classes A, B, and C conceptually indicates the difference in bandwidth. That is, it indicates that transmission class A, transmission class B, and transmission class C have larger bandwidths in that order. However, the magnitude of the height difference shown does not correspond to the magnitude of the bandwidth difference. The frame frequency of the camera 112d (camera adapter 120d) is half the frame frequency of the other cameras 112a to 112c and 112e to 112f, and is always assigned a fixed transmission class C band. In FIG. 9, one frame of imaging data of the camera 112d whose transmission class is C is transferred over two frame periods of the other cameras.

In frame #1, the imaged data of the cameras 112f and 112b are classified into the transmission class A, and the imaged data of the cameras 112e, 112c and 112a are classified into the transmission class B. Also, since the camera adapter 120d connected to the camera 112d is set to the transmission class C, the transmission class C is selected for the imaging data of the camera 112d. Transmission of imaging data of the cameras 112f and 112e by the camera adapters 120f and 120e is the same as in the first embodiment (FIG. 7). Since the imaged data of the camera 112d is classified into transmission class C, the camera adapter 120d transmits the packet of the imaged data of the camera 112d in the transmission class C band immediately after the start of the frame. The transmission class of the header of the image data of the camera 112d is set to C, the transmission class of the header of the last packet is set to 1, and the transmission class of the headers of the other packets is set to 0. Also, the count-up value of the start timer 135 of the camera adapter 120d is set to 0, and the count-up value of the drop timer 136 is set to the value immediately before the end of the two frame periods.

In the camera adapter 120c, the transfer of the imaging data classified into the transmission class B (the imaging data of the camera 112e) ends at time Tc1. The camera adapter 120c determines whether the transmission can be completed within the remaining time of frame #1 when the imaging data of the camera 112c is transmitted in the band assigned to transmission class B. Here, it is determined that the transmission of the imaging data of the camera 112c can be completed, and the camera adapter 120c starts transmitting the imaging data of the camera 112c downstream in the band assigned to transmission class B.

In the camera adapter 120b, the transfer of the imaging data classified into the transmission class A (the imaging data of the camera 112f) ends at time Tb1. When the camera adapter 120b transmits the imaging data of the camera 112b in the band assigned to the transmission class A, the camera adapter 120b determines whether the transmission is completed within the remaining time of the frame #1. Here, it is determined that the transmission of the imaged data of the camera 112b is completed, and the camera adapter 120b starts transmitting the imaged data of the camera 112c in the band assigned to the transmission class A. The transmission of the imaging data of the camera 112c is completed at time Tb1E.

In the camera adapter 120a, transfer of imaged data (imaged data of the cameras 112e and 112b) classified into transmission class B ends at time Ta1. The camera adapter 120a determines whether the transmission is completed within the remaining time of the frame #1 when the imaging data of the camera 112a is transmitted in the band assigned to the transmission class B. Here, it is determined that the transmission of the imaging data of the camera 112a cannot be completed, and the camera adapter 120a discards the imaging data of the camera 112a without transmitting it. The imaging data of the camera 112d is transmitted over the periods of frames #1 and #2 in the transmission class C band.

As described above, in the example of FIG. 9, in frame #1, it is determined that the imaging data of the camera 112a located at the most downstream position in transmission class B cannot be transmitted within the frame, and is discarded. On the other hand, all the imaging data of the camera 112 classified into transmission class A could be transmitted to the image computing server 200 .

In frame #2, the imaged data of cameras 112e and 112c are classified into transmission class A, the imaged data of cameras 112f, 112b, and 112a are classified into transmission class B, and the imaged data of camera 112d are classified into transmission class C. . The imaging data of the cameras 112f and 112e are respectively transmitted from the camera adapters 120f and 120e after the start of frame #2 as in the case of frame #1. Also, the transmission of imaged data from the camera 112d continues from frame #1. At time Td12E, the camera adapter 120d completes transmission of the imaging data of the camera 112d.

In the camera adapter 120c, the transfer of the imaging data classified into the transmission class A (the imaging data of the camera 112e) ends at time Tc2. When the camera adapter 120c transmits the imaging data of the camera 112c in the band assigned to the transmission class A, it determines whether the transmission is completed within the remaining time of frame #2. Here, it is determined that the transmission of the imaged data of the camera 112c can be completed, and the camera adapter 120c starts transmitting the imaged data of the camera 112c in the band assigned to the transmission class A. Transmission of the camera 112c by the camera adapter 120c is completed at time Tc2E before reaching time TE2 immediately before the end of the frame.

In the camera adapter 120b, the transfer of the imaging data classified into transmission class B (the imaging data of the camera 112f) ends at time Tb2. When the camera adapter 120b transmits the imaging data of the camera 112b in the band assigned to the transmission class B, the camera adapter 120b determines whether the transmission is completed within the remaining time of frame #2. Here, it is determined that transmission of the imaged data of the camera 112b can be completed, and the camera adapter 120b starts transmission of the imaged data of the camera 112b in the band assigned to transmission class B.

In the camera adapter 120a, the transfer of the imaged data (imaged data of the cameras 112f and 112b) classified into the transmission class B ends at time Ta2. When the camera adapter 120a transmits the imaging data of the camera 112a in the band assigned to the transmission class B, the camera adapter 120a determines whether the transmission is completed within the remaining time of the frame #1. Here, it is determined that the transmission of the imaging data of the camera 112a cannot be completed, and the camera adapter 120a discards the imaging data of the camera 112a without transmitting it.

As described above, in frame #2, only the imaging data of the camera 112a located at the most downstream position of transmission class B cannot be transmitted within the frame, so that it is discarded without being transmitted. All the imaging data of the camera 112 classified as transmission class A could be transmitted to the image computing server 200 . The imaging data of the camera 112d is transferred in the transmission class C band continuously from frame #1.

As described above, according to the second embodiment, even when transmission classes with different frame frequencies are provided, it is possible to allocate a band for each transmission class and perform transmission. As described above, according to each of the above-described embodiments, even if the amount of captured image data temporarily exceeds the bandwidth of the transmission path and greatly increases, missing captured image data is concentrated in a specific camera close to the image computing server 200. no longer. In addition, since a transmission class for imaging data with a large amount of data is secured, important imaging data can be transmitted to the image computing server 200 more reliably.

As described in the first embodiment, normal transmission control and transmission control according to the second embodiment may be switched. In this case, even in normal transmission control, only packets of transmission class C may be transmitted in an independent band. Alternatively, in normal transmission control, a packet of transmission class C may be distributed to each frame and transmitted. For example, in the example of FIG. 9, under normal transmission control, a packet of transmission class C may be transmitted in half in frame #1 and half in frame #2.

Further, in the above embodiment, an example was shown in which image data from an imaging device is used as data to be transmitted by a plurality of data transmission devices connected in a daisy chain, but the data to be transmitted is limited to this. isn't it. For example, the data to be transmitted may be acoustic data. Alternatively, the camera adapter may have a function of extracting the foreground image, and the foreground image may be the data to be transmitted.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

112: camera, 120: camera adapter, 121: transmission data generator, 122, 123: port, 130: transmission/reception processing unit, 131: reception data holding unit, 132: reception processing unit, 133: transmission processing unit, 134: transmission Data holding unit, 135: start timer, 136: drop timer

Claims (13)

A data transmission device connected to a first external device and a second external device by a daisy chain,
generating means for generating packets from data to be transmitted;
setting means for setting, to the packet generated by the generation means, a band selected based on the data amount of the data to be transmitted from among a plurality of bands obtained by dividing the data transmission band in the daisy chain into different bands; and,
transmitting means for performing transmission in the plurality of bands in parallel by transmitting packets at a frequency corresponding to the plurality of bands, wherein the packet received from the first external device and the packet generated by the generation means; transmitting means for transmitting the packets to the second external device in a band set for each packet. 2. The method according to claim 1, wherein said transmission means transmits said received packets to said second external device in the order in which they were received from said first external device in each of said plurality of bands. data transmission equipment. The transmitting means transmits the generated packet in the set band after finishing transmission of the packet set in the same band as the band set in the generated packet among the received packets. 3. The data transmission device according to claim 1, wherein the transmission is performed. A transmission flag indicating that the last packet among the packets transmitted in each of the plurality of bands is the last packet is set to ON;
The generating means sets on the transmission flag of the last packet generated from the data to be transmitted,
When a packet having the same bandwidth as that of the generated packet among the received packets and having the transmission flag set to ON is received, the transmission means transmits the transmission flag of the packet. 4. The data transmission apparatus according to claim 3, wherein the packet is transmitted after setting to off, and then transmission of the generated packet is started. 5. The data transmission apparatus according to claim 4, wherein bit 0 of a flag field of an IP header attached to the packet is used as said transmission flag. 6. The setting means selects a wider band from the plurality of bands as the amount of data to be transmitted is larger, and sets the selected band to the generated packet. The data transmission device according to . 7. A part of a fragment offset field or a part of a service type field of an IP header attached to a packet is used to indicate a band set for the packet. 1. The data transmission device according to claim 1. 8. The data transmission device according to any one of claims 1 to 7, wherein the data to be transmitted is imaging data obtained from an imaging device connected to the data transmission device. the plurality of bands includes a predetermined band for transferring data in a second frame period different from a first frame period in which data is transferred by another device connected to the daisy chain;
When the data transmission device is set to transfer data in the second frame cycle, the setting means sets the predetermined band in the generated band regardless of the amount of data to be transmitted. 9. The data transmission device according to any one of claims 1 to 8, wherein the data is set in a packet. A data transmission system comprising a plurality of data transmission devices connected by a daisy chain and a plurality of cameras connected to each of the plurality of data transmission devices,
each of the plurality of data transmission devices,
generating means for generating packets from data to be transmitted;
A band selected based on a data amount of the data to be transmitted from among a plurality of bands obtained by dividing a band for data transmission in the daisy chain into a plurality of different bands is set in the packet generated by the generating means. setting means;
transmitting means for performing transmission in the plurality of bands in parallel by transmitting packets at a frequency corresponding to the plurality of bands, wherein the packet received from the upstream data transmission device connected by the daisy chain; and transmitting means for transmitting the packets generated by the generating means to downstream data transmission devices connected by the daisy chain in a band set for each packet. . The transmission means transmits packets in a first mode in which packets are transmitted in parallel in the plurality of bands, and in a second mode in which packets are transmitted using the daisy chain data transmission band without distinguishing between the plurality of bands. It works in 2 modes,
The apparatus further comprises determining means for determining which of the first mode and the second mode to use based on the state of the band set by the setting means of the plurality of data transmission devices. 11. The data transmission system according to claim 10, wherein A control method for a data transmission device connected by a daisy chain to a first external device and a second external device, comprising:
a generation step of generating packets from data to be transmitted;
A band selected based on the amount of data to be transmitted from among a plurality of bands obtained by dividing a band for data transmission in the daisy chain into a plurality of different bands is set in the packet generated by the generating step. a setting process;
A transmission step of performing transmission of the plurality of bands in parallel by transmitting packets at a frequency corresponding to the plurality of bands, wherein the packet received from the first external device and the packet generated by the generation step and a transmission step of transmitting the packets to the second external device in a band set for each packet. A program for causing a computer to function as each means of the data transmission device according to any one of claims 1 to 9.

Images