JP2022028499A - Communication device, communication method, and program - Google Patents

Abstract

To improve the reliability of time synchronization between a plurality of devices.SOLUTION: A communication device comprises: first correction means that synchronizes first time information with synchronization information received through a first communication path; second correction means that synchronizes second time information with synchronization information received through a second communication path; synchronization means that, when the synchronization information through the first communication path or the synchronization information through the second communication path is not received, synchronizes the time information corresponding to the communication path through which the synchronization information is not received with the other time information; first transmission means that transmits synchronization information based on the first time information to a subsequent stage device through the first communication path; and second transmission means that transmits synchronization information based on the second time information to the subsequent stage device through the second communication path.SELECTED DRAWING: Figure 2

Description

The present invention relates to communication devices, communication methods and programs.

PTP (Precision Time Protocol) is known as one of the time synchronization methods using a network in order to realize synchronous shooting by a plurality of devices. In the synchronization processing using this PTP (PTP synchronization processing), the master device having an accurate time transmits a PTP packet including information indicating the time (time information), and the slave device receiving the PTP packet is a PTP. Synchronize the time according to the time information in the packet.

Further, as a connection method of a plurality of devices constituting a network, a connection method called a daisy chain in which each communication device is connected in series is known. In this daisy chain, each communication device has a plurality of connectors, and a connection by the daisy chain can be established by connecting to a different communication device via each connector.

Further, as an example of the time synchronization control method, Patent Document 1 discloses a technique of using a network using PTP. In the technique disclosed in Patent Document 1, a network is configured by connecting a GM (Grand Master) device and a plurality of BC (Brandary Clock) devices in a daisy chain. The GM device receives accurate time information from the global positioning system GNSS (Global Navigation Satellite System) and distributes it to the BC device. The BC device corrects the time information received from the GM device to the time of the GM device and distributes it to the BC device to be connected next. This realizes time synchronization of each device connected to the network.

Japanese Unexamined Patent Publication No. 2019-158538

However, in the technique disclosed in Patent Document 1, synchronization information is transmitted via a plurality of routes, but in a device on the downstream side (post-stage) of the route where the link failure has occurred, the device has the link failure via the path where the link failure has occurred. There was a problem that the synchronization could not be maintained.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to improve the certainty of time synchronization between a plurality of devices.

In order to solve the above-mentioned problems, according to a certain aspect of the communication device according to the present invention, a first correction means for synchronizing the first time information with the synchronization information received via the first communication path, and the first correction means. The second correction means for synchronizing the second time information with the synchronization information received via the second communication path, and the synchronization information via the first communication path or the second communication path. When the synchronization information is not received, the synchronization means for synchronizing the time information corresponding to the communication path for which the synchronization information is not received with the other time information, and the synchronization information based on the first time information are the first. A first transmission means for transmitting to a subsequent device via the communication path 1 and a second transmission means for transmitting synchronization information based on the second time information to the device at the subsequent stage via the second communication path. A communication device comprising a transmission means is provided.

A block diagram showing a configuration example of a shooting system. A block diagram showing a configuration example of the network control unit. A flowchart showing an example of synchronization processing between a time server and a sensor system. A flowchart showing an example of time synchronization processing between sensor systems. The figure which shows the example of the sequence of the synchronization packet in the PTP synchronization processing. A flowchart showing an example of time synchronization processing at the time of a failure. A flowchart showing an example of time synchronization processing at the time of a failure.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. The embodiments described below are examples of means for realizing the present invention, and should be appropriately modified or modified depending on the configuration of the apparatus to which the present invention is applied and various conditions. Is not necessarily limited to the embodiment of. Moreover, not all combinations of features described in the embodiments are essential to the solution of the present invention.

(Embodiment 1)
(Synchronous shooting system configuration)
FIG. 1 is a block diagram showing a configuration example of the synchronous shooting system (hereinafter, simply referred to as a shooting system) 10 of the first embodiment.
The photographing system 10 can generate an image of a virtual viewpoint (virtual viewpoint) in a stadium, a concert hall, or the like.
The photographing system 10 includes a first time server 11a, a second time server 11b, a hub 12, a control terminal 13, an image processing server 14, a user terminal 15, and sensor systems 20a, 20b, ... 20z. I have. The hub 12 and each of the sensor systems 20a to 20z have two daisy-chain communication paths (first communication path, second communication path, ...) Consisting of communication paths 16a, 16c, ..., And communication paths 16b, 16d, .... It is connected by the communication path). The number of communication paths may be three or more.

In the present embodiment, unless otherwise specified, the 26 sets of sensor systems from the sensor system 20a to the sensor system 20z may be shown as the sensor system 20 without distinction. Similarly, the processing unit in each sensor system 20 may be shown without distinction unless otherwise specified. The number of sensor systems 20 is shown as 26 sets of 20a to 20z, but this is just an example, and the number of sensor systems 20 is not limited to this example. Further, the plurality of sensor systems 20 do not have to have the same configuration, and may have different configurations, for example. In the present embodiment, unless otherwise specified, the word "image" is used as including the concepts of moving images and still images. That is, it is assumed that the photographing system 10 of the present embodiment can process both still images and moving images.

The first time server 11a and the second time server 11b have a function of distributing time information, and distribute time information synchronized with GNSS (Global Navigation Satellite System) 17 to each sensor system 20. The first time server 11a distributes time information via the first communication path including the communication path 16a. Further, the second time server 11b distributes the time information via the second communication path including the communication path 16b. It should be noted that either one of the time servers may distribute the time information via both the first communication path and the second communication path. Further, the first time server 11a may distribute the time information via the second communication path, or the second time server 11b may distribute the time information via the first communication path. In addition, in case of an unexpected situation such as a failure in one of the two time servers, two are arranged in the system. In the following description, unless otherwise specified, the two time servers 11a and 11b are collectively described as the time server 11.

The hub 12 distributes PTP (Precision Time Protocol) synchronization packets including time information delivered by the first time server 11a and the second time server 11b and image data transmitted by the sensor system 20 to the image processing server 14 to each destination. .. The hub 12 has a routing function between a protocol that communicates with the sensor system 20 and a network that communicates with the control terminal 13 and the like.
The control terminal 13 communicates with each device constituting the photographing system 10 through a network via the hub 12, and executes operation state management, parameter setting control, and the like. As the network, an IEEE (Ethernet, registered trademark, hereinafter abbreviated) IEEE standard compliant Gigabit Ethernet or 10 Gigabit Ethernet may be used. Alternatively, a network may be configured by combining an interconnect technology such as Infiniband, an industrial Ethernet, or the like. Further, the present invention is not limited to these examples, and other types of networks may be used.

Each sensor system 20 includes a camera 21 for photographing a subject and a camera adapter 22 for controlling the operation of the camera 21. Each camera 21a to 21z shoots a subject from a plurality of directions. Hereinafter, each camera 21 will be described as the same, but the performance and model may be different for each camera 21. The configuration of each sensor system is not limited to this. For example, a voice device such as a microphone, a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), a pan head for controlling the orientation of a camera, or the like may be included. In this case, the sensor system 20 may have a function of storing the image data in the storage device in correspondence with the time information. Further, for example, the sensor system 20 may be composed of one camera adapter and a plurality of cameras, or may be composed of one camera and a plurality of camera adapters.

The plurality of sensor systems 20 are connected by a daisy chain communication path. This daisy-chain connection form reduces the number of connection cables and saves power in wiring work, for example, in the transmission of image data of large-capacity captured images with high resolution such as so-called 4K and 8K and high frame rate. Has the effect of being able to. The connection form of each sensor system 20 or the like may be, for example, a star-type network in which each sensor system 20 is connected to the hub 12 and data is transmitted / received between the sensor systems 20 via the hub 12. Further, FIG. 1 shows a configuration in which all of the sensor systems 20 are daisy-chained, but the present invention is not limited to this. For example, a plurality of sensor systems 20 may be divided into several groups, and the sensor systems 20 may be daisy-chained within the divided groups. Such a configuration is particularly effective in a wide place such as a stadium. For example, a stadium may be composed of a plurality of floors, and a sensor system 20 may be installed on each floor. In this case, each sensor system 20 can transmit image data to the image processing server 14 for each floor or every half lap of the stadium. This makes it possible to simplify the installation and make the system more flexible even in a place where wiring for connecting all the sensor systems 20 with one daisy chain is difficult.

Further, the camera adapters 22a to 22z include a camera control unit 23, an image processing unit 24, and a network control unit 25.
The camera control unit 23 executes control of the camera 21, acquisition of captured images, provision of synchronization signals, time setting, and the like.
The image processing unit 24 performs image processing on the image pickup data captured by the camera 21. Specifically, for example, a foreground image and a background image are output from the captured data (transmitted to the image processing server 14). As a result, the image processing server 14 can generate a virtual viewpoint image based on the foreground image and the background image taken from a plurality of viewpoints. There may be a camera 21 that outputs a foreground image separated from a captured image and does not output a background image.

The network control unit 25 transmits the image-processed data to the image processing server 14 via the hub 12 connected by the daisy chain. Further, the network control unit 25 operates as a PTP slave to synchronize the clock (time information) inside the device with the time server 11 which is a GM (Grand Master) device. Alternatively, the network control unit 25 operates as a PTP master to synchronize the clock (time information) inside the device of the other sensor system 20. The network control unit 25 uses the synchronized time (clock) and reference signal to generate a shooting timing (control clock) and provides it to the camera 21.

The image processing server 14 processes the data acquired from the sensor system 20.
First, the image processing server 14 reconstructs the image pickup data acquired from the sensor system 20 to convert the data format, and then stores the image according to the identifier of the camera, the data type, the frame number, and the like. Further, the image processing server 14 reads out the imaging data corresponding to the viewpoint designated from the control terminal 13 from the stored information, performs rendering processing, and generates an image of the virtual viewpoint. The control terminal 13, the sensor system 20, and the user terminal 15 may have at least a part of the functions of the image processing server 14.

The user terminal 15 receives the rendered image from the image processing server 14 and displays it. As a result, the user who operates the user terminal 15 can view the image of the designated viewpoint (virtual viewpoint). That is, the image processing server 14 generates a virtual viewpoint image (virtual viewpoint content) based on the captured image (image of a plurality of viewpoints) taken by the plurality of sensor systems 20 and the viewpoint information. In the present embodiment, the virtual viewpoint content is generated by the image processing server 14, but the present invention is not limited to this. That is, the virtual viewpoint content may be generated by the control terminal 13 or the user terminal 15.

(Details of network control unit)
FIG. 2 is a block diagram showing a detailed configuration example of the network control unit of the camera adapter.
The path between the camera adapters 22 daisy-chained by the communication paths 16a, 16c, ... Is defined as the first communication path, and the path between the camera adapters 22 daisy-chained by the communication paths 16b, 16d, ... Is defined as the first communication path. It is defined as the second communication path.
As a specific example, the camera adapter 22 that has received a PTP synchronization packet (hereinafter, simply referred to as a PTP packet) via the communication path 16a executes an appropriate synchronization process by the network control unit 25 to set the communication path 16c. Send a PTP packet via. Similarly, the camera adapter 22 that has received the PTP packet via the communication path 16b executes an appropriate synchronization process by the network control unit 25, and transmits the PTP packet via the communication path 16d. That is, the synchronization processing executed in the first communication path via the daisy chain (16a, 16c) and the second communication path via the daisy chain (16b, 16d) is executed independently, and each communication path has a redundant configuration. It has become. This is because when some functions are shared between the first communication path and the second communication path, synchronization processing cannot be performed between the time server 11 and the sensor system 20 and between the sensor system 20 due to a failure of the shared function or the like. This is to prevent this.
Hereinafter, unless otherwise specified, the functional blocks 31a to 38a and 41a to 48a constituting the network control unit 25 of the respective sensor systems 20a to 20z are simply referred to as 31 to 38 and 41 to 48 without distinction. ..

The network control unit 25 includes two systems of control units 251, 252. Each control unit 251 (252) includes a communication port 31 (41), a failure recovery detection unit 32 (42), a PTP packet detection unit 33 (43), a clock control unit 34 (44), and a clock 35 (45). ) And the protocol processing unit 36 (46). Further, each control unit 251 (252) includes a PTP packet detection unit 37 (47) and a communication port 38 (48).
Each control unit 251 (252) sends and receives PTP packets for time synchronization of the sensor system 20 via the communication ports 31a, 38a, 41a, and 48a. Further, the image pickup data taken by the camera 21 and processed by the image processing unit 24 is also transmitted and received via the communication ports 31a, 38a, 41a and 48a in the same manner as the PTP packet. Similarly, communication packets such as TCP / UDP / IP are also transmitted / received using the communication ports 31a, 38a, 41a, 48a. Here, TCP is an abbreviation for Transmission Protocol, UDP is an abbreviation for User Datagram Protocol, and IP is an abbreviation for Internet Protocol.

The failure recovery detection unit 32 (42) detects a link failure such as a cable disconnection and recovery from a link failure, and notifies the clock control unit 34 (44) of the detection result to be described later.
The PTP packet detection units 33, 37 (43, 47) detect the PTP packet received from the communication ports 31, 38 (41, 48), and notify the clock control unit 34 (44) of the detection result. Further, when the PTP packet detection unit 33 (43) does not receive the PTP packet for a certain period of time or a certain number of PTP packets, the PTP packet detection unit 33 (43) notifies the clock control unit 34 (44). Similarly, when the PTP packet detection unit 37 (47) does not receive a certain number of PTP packets for a certain period of time, the PTP packet detection unit 37 (47) notifies the clock control unit 34 (44). In this example, PTP packet detection units 33, 37, 43, 47 are provided for each communication port 31, 38, 41, 48, but even if one PTP packet detection unit realizes the same function. good.

The clock control unit 34 (44) holds the time information of the received PTP packet and corrects the clock 35 (45) described later to the time of the time server 11. Further, when the clock control unit 34 (44) receives a link failure detection notification from the failure recovery detection unit 32 (42) or a notification that the PTP packet is not received from the PTP packet detection unit 33 (43), the clock control unit 34 (44) receives a notification. Acquires the time information of the clock synchronized with the time server 11. Here, the communication path synchronized with the time server 11 is a communication path without a communication failure or the like. When the time information is acquired, the clock control unit 34 (44) corrects the clock 35 (45) corresponding to the communication path in which the failure has occurred.
Specifically, for example, when a failure occurs in the first communication path, the clock control unit 34 is notified that the link failure is detected from the failure recovery detection unit 32 or that the PTP packet from the PTP packet detection unit 33 is not received. Will be sent to. Upon receiving any of the notifications, the clock control unit 34 acquires information indicating the time of the clock 45 from the clock control unit 44, and synchronizes the time of the clock 35 with the time of the clock 45 according to the acquired information.

The clock 35 (45) holds the time used in the synchronization processing corresponding to each communication path. The time held by the clock 35 (45) is synchronized with the time of the time server 11 according to the control from the clock control unit 34 (44) according to the PTP packet received at regular intervals.
In this embodiment, a case where one clock 35, 45 is provided for each communication path (first communication path, second communication path) will be described, but the number and arrangement of clocks are not limited to this example. A configuration having two clocks for each communication path, for example, a configuration having clocks for each communication port 31a, 38a, 41a, 48a may be used. In this case, synchronization is executed between the clocks corresponding to the communication ports 31a and 38a of the first communication path and between the clocks corresponding to the communication ports 41a and 48a of the second communication path. Further, the clock may be shared by each communication path. For example, as a configuration in which a clock is provided for each of the first communication port 31a and the second communication port 41a, and one common clock (common clock) is provided for the third communication port 38a and the fourth communication port 48a. May be good. In this case, the time synchronization process is executed between the clock corresponding to the first communication port 31a and the common clock, or between the clock corresponding to the second communication port 41a and the common clock.

The protocol processing unit 36 executes synchronization processing (PTP synchronization processing) using PTP.
The PTP synchronization process defined in IEEE1588-2008 has three operation modes: TC (Transient Clock), BC (Brandary Clock), and OC (Ordinary Clock). In this embodiment, PTP synchronization processing is executed by, for example, BC.
In the synchronization process by BC, the device to be synchronized first corrects the time information of its own device by using the time information received from the time server which is the GM as the slave device. When the time information of the own device is corrected, the device, as the master device, transmits a PTP packet to the device in the subsequent stage and executes the synchronization process. In the present embodiment, the protocol processing unit 36 (46) corrects the time information (clock 35 (45)) of its own device from the received PTP packet as a slave device in BC. After that, the protocol processing unit 36 (46) transmits a PTP packet to the sensor systems 20b to 20z in the subsequent stage as a master device in BC. Although the clock control unit 34 (44) executes the time correction, the protocol processing unit 36 (46) may correct the time of the clock 35 (45).

(Time synchronization processing)
FIG. 3 is a flowchart showing an example of time synchronization processing (PTP synchronization processing) between the time server 11 and the sensor system 20.
In this photographing system 10, after the GMC (Grand Master Clock) is determined to be the time server 11 and the PTP slave is determined to be the sensor system 20, the process of FIG. 3 is started (S1). The transmission / reception processing of the synchronization packet and the PTP synchronization processing are executed by the protocol processing unit 36 (46) in the sensor system 20. Hereinafter, the protocol processing unit 36 will be described as processing the synchronous packet, but the protocol processing unit 46 is also executing the same processing.

First, in S2, the protocol processing unit 36 determines whether or not the sensor system 20 has received the synchronization packets Sync and Follow_Up from the time server 11. When any of the synchronization packets is received, the protocol processing unit 36 proceeds to S3, and when none of the synchronization packets is received, the protocol processing unit 36 repeats the processing of S2 and waits for the reception of the synchronization packet.
In S3, the protocol processing unit 36 generates a synchronization packet Delay_Request and transmits it to the time server 11. When the transmission of the synchronization packet is completed, the protocol processing unit 36 proceeds to S4.

In S4, the protocol processing unit 36 determines whether or not the synchronization packet Delay_Response has been received from the time server 11. The protocol processing unit 36 proceeds to S6 when the synchronization packet is received, and proceeds to S5 when the synchronization packet is not received.
In S5, the protocol processing unit 36 does not receive the synchronization packet Delay_Response from the time server 11 and determines whether or not a certain period of time has elapsed. The protocol processing unit 36 returns to S2 when a certain time has elapsed, and returns to S4 when a certain time has not passed.

In S6, the protocol processing unit 36 calculates the time difference between the time of the time server 11 and the time of the sensor system 20 from the synchronization packet exchanged with the time server 11 and determines the correction amount. When the time correction amount is determined, the protocol processing unit 36 proceeds to S7. A specific method for determining the correction amount will be described later with reference to FIG. In this example, the protocol processing unit 36 determines the correction amount, but the clock control unit 34 may determine the correction amount.
In S7, the clock control unit 34 corrects the clock 35 based on the correction amount determined by the protocol processing unit 36 in S6. When this correction process is completed, the clock 35 synchronizes with the time server 11. When the correction of the clock 35 is completed, the protocol processing unit 36 proceeds to S8. If the amount of correction at one time is large, the operation of the entire sensor system 20 may become unstable. Therefore, when the correction amount is larger than a predetermined threshold value, the clock correction (clock synchronization processing) may be performed stepwise over a predetermined time. Alternatively, the amount of correction at one time may be limited.

In S8, the protocol processing unit 36 determines whether or not to continue the operation of the sensor system 20. The case where the operation of the sensor system 20 is not continued means that, for example, the control terminal 13 stops the synchronization processing of the entire sensor systems 20a to 20z, or the synchronization processing is performed only for the specified one of the sensor systems 20a to 20z. For example, when stopping.
If the operation of the sensor system 20 is to be continued, the protocol processing unit 36 returns to S2 to continue the processing, and if not, the processing of FIG. 3 is terminated.

FIG. 4 is a flowchart showing an example of time synchronization processing between sensor systems.
Hereinafter, synchronization processing between the sensor system 20a and the sensor system 20b will be described as an example. In the sensor systems 20a and 20b, the GMC is determined to be the sensor system 20a and the PTP slave is determined to be the sensor system 20b. After that, the protocol processing unit 36a of the sensor system 20a starts the PTP synchronization processing as the master device, so that the processing of FIG. 4 is started (S11).

First, in S12, the protocol processing unit 36a transmits synchronization packets Sync and Follow_Up to the sensor system 20b at regular intervals. When the transmission of the synchronization packet is completed, the protocol processing unit 36a proceeds to S13.
In S13, the protocol processing unit 36a determines whether or not the synchronization packet Delay_Request has been received from the sensor system 20b. The protocol processing unit 36a proceeds to S15 if the synchronization packet is received, and proceeds to S14 if the synchronization packet is not received.
In S14, the protocol processing unit 36a does not receive the synchronization packet Delay_Request from the sensor system 20b, and determines whether or not a certain period of time has elapsed. When the fixed time has elapsed, the protocol processing unit 36a returns to S12, and if the fixed time has not passed, returns to S13 and waits for the reception of the synchronization packet.

In S15, the protocol processing unit 36a transmits a synchronization packet Delay_Response to the sensor system 20b. When the transmission of the transmission packet is completed, the protocol processing unit 36a proceeds to S16.
In S16, the protocol processing unit 36a determines whether or not the sensor system 20 continues to operate. The case where the operation of the sensor system 20 is not continued means that, for example, the control terminal 13 stops the synchronization processing of the entire sensor systems 20a to 20z, or the synchronization processing is performed only for the specified one of the sensor systems 20a to 20z. For example, when stopping.
If the operation of the sensor system 20 is to be continued, the protocol processing unit 36a returns to S12 to continue the processing, and if not, the processing of FIG. 4 is terminated.

FIG. 5 shows an example of a sequence of synchronization packets of a GMC (Grand Master Clock) and a PTP slave device in PTP synchronization processing.
FIG. 5 shows, as an example, transmission / reception of a synchronization packet between a time server 11 which is a GMC master device (Mst) and a sensor system 20a and a sensor system 20b which are BC (Brandary Clock) devices.
First, transmission / reception of synchronization packets between the time server 11 which is a GMC master device (Mst) and the sensor system 20a which is a PTP slave device (Slv) will be described.

(Synchronized packet Sync)
The time server 11 which is a master device transmits a synchronization packet Sync (S24a) to the sensor system 20a at regular intervals.
The sensor system 20a, which is a slave device, holds the time t2 when the Sync from the time server 11 is received.
(Synchronous packet Follow_Up)
The time server 11 transmits a synchronization packet Follow_Up (S25a) including the time t1 at which the synchronization packet Sync is transmitted to the sensor system 20a. Upon receiving the synchronization packet Follow_Up, the sensor system 20a holds the time t1 in the packet. Hereinafter, a method of including the transmission time of the synchronization packet Sync in the synchronization packet Follow_Up and transmitting the synchronization packet will be described as an example. The transmission time t1 of the synchronization packet Sync may be included in the synchronization packet Sync itself for transmission instead of the synchronization packet Follow_Up.

(Synchronous packet Delay_Request)
The sensor system 20a transmits a synchronization packet Delay_Request (S26a) to the time server 11. At this time, the sensor system 20a holds the transmission time t3 of Delay_Request.
(Synchronous packet Delay_Response)
Upon receiving the synchronization packet Delay_Request from the sensor system 20a, the time server 11 transmits the synchronization packet Delay_Response (S27a) to the sensor system 20a. At this time, the time server 11 includes the reception time t4 of the synchronization packet Delay_Request in the synchronization packet Delay_Response and transmits it to the sensor system 20a. Upon receiving the synchronization packet Delay_Response, the sensor system 20a holds the time t4 in the packet.

The sensor system 20a calculates the average transmission delay time from the transmission / reception times t1 to t4 of the synchronization packets S24a to S27a, and corrects the time of the clock 35a.
The average transmission delay time is calculated by the following formula.
Average transmission delay time = {(t2-t1) + (t4-t3)} / 2

Next, transmission / reception of a synchronization packet between the sensor system 20a which is a PTP master device (Mst) and the sensor system 20b which is a PTP slave device (Slv) will be described.
The sensor system 20a, which is a BC device, operates as a PTP master (Mst) and executes synchronization packets with the sensor system 20b, which is a PTP slave device (Slv).
The transmission and reception of the synchronization packet is the same as the operation between the time server 11 and the sensor system 20a described above, and the synchronization packets S24b to S27b correspond to the synchronization packets S24a to S27a described above.
The sensor system 20b calculates the average transmission delay time from the transmission / reception times of the synchronized packets S24b to S27b and corrects the time of the clock 35b in the same manner as the operation of the sensor system 20a when operating as the slave device described above.

6 and 7 are flowcharts showing an example of time synchronization processing when a failure occurs.
Hereinafter, an example of PTP synchronization processing between the time server and the sensor system will be described in the present embodiment, but the present invention is not limited to this, and can also be applied to PTP synchronization processing between sensor systems. Further, the present invention is applied even when the failure occurs only in the first communication path, only in the second communication path, or occurs in both the first communication path and the second communication path. can.

The process shown in FIGS. 6 and 7 is started in a state where the entire photographing system 10 is PTP synchronized (S31). Hereinafter, the processing on the control unit 251 side will be described as an example, but the same processing is also executed on the control unit 252 side.
First, in S32, the protocol processing unit 36 determines whether or not the failure recovery detection unit 32 has detected a link failure in each communication path. A link failure is a state in which a link between devices is down due to, for example, a disconnection of a network cable constituting each communication path or a failure of a functional block related to communication in each communication path. When the failure recovery detection unit 32 detects a link failure, the protocol processing unit 36 proceeds to S41 in FIG. 7, and if no link failure is detected, proceeds to S33.

In S33, the protocol processing unit 36 determines whether the PTP packet detection unit 33 has not received the PTP packet for a certain period of time, or has not received a certain number of PTP packets. The protocol processing unit 36 proceeds to S41 if any of the conditions is satisfied, and proceeds to S34 in other cases.
In S34, the protocol processing unit 36 that operates as a PTP slave determines whether or not the synchronization packets Sync and Follow_Up have been received from the time server 11. The protocol processing unit 36 proceeds to S35 when the synchronization packet is received, and returns to S32 when the synchronization packet is not received.

In S35, the protocol processing unit 36 that operates as a PTP slave transmits a synchronization packet Delay_Request to the time server 11. When the transmission of the synchronization packet is completed, the protocol processing unit 36 proceeds to S36.
In S36, the protocol processing unit 36 determines whether or not the synchronization packet Delay_Response has been received from the time server 11. The protocol processing unit 36 proceeds to S38 when the synchronization packet Delay_Response is received, and proceeds to S37 if it has not been received.
In S37, the protocol processing unit 36 determines whether a certain period of time has elapsed without receiving the synchronization packet Delay_Response from the time server 11. If the fixed time has not elapsed, the protocol processing unit 36 returns to S36 and repeats the determination of whether or not the synchronization packet has been received. If a certain period of time elapses without receiving the synchronization packet, the protocol processing unit 36 discards the result of transmission / reception of the synchronization packet so far and returns to S32.

In S38, the protocol processing unit 36 calculates the correction amount of the clock 35 based on the time information obtained by sending and receiving the synchronization packet to and from the time server 11. When the calculation of the correction amount is completed, the protocol processing unit 36 proceeds to S39.
In S39, the clock control unit 34 corrects the time of the clock 35 with the correction amount calculated in S38. When the clock control unit 34 completes the correction of the clock 35, the protocol processing unit 36 proceeds to S40.
In S40, the protocol processing unit 36 determines whether or not to continue the operation of the sensor system 20. Specifically, for example, when the control terminal 13 stops the synchronization processing of the entire sensor systems 2a to 20z, or when only the specified one of the sensor systems 20a to 20z stops the synchronization processing. , It is determined that the operation of the sensor system 20 is not continued.
If the operation of the sensor system 20 is to be continued, the protocol processing unit 36 returns to S32 to continue the processing, and if not, the processing is terminated.

In S41 of FIG. 7, the failure recovery detection unit 32 notifies the clock control unit 34 that a link failure has occurred or that the PTP packet is not received. When the notification is completed, the failure recovery detection unit 32 proceeds to S42.
In S42, the protocol processing unit 36 determines whether or not the failure recovery detection unit 32 has detected that the link failure of both the first communication path and the second communication path has been resolved and the PTP packet can be transmitted / received. judge. If the failure recovery detection unit 32 detects the recovery of the link failure, the protocol processing unit 36 proceeds to S43, and if not, proceeds to S44.
In S43, the protocol processing unit 36 determines whether or not the synchronization packets Sync and Follow_Up have been received from the time server 11. When the protocol processing unit 36 receives the synchronization packet, it returns to S35 and executes the subsequent processing. That is, when the reception of the synchronization packet is resumed via the communication path in which the link failure has occurred, the synchronization processing of the clock 45 to the clock 35 is terminated. If the synchronization packet has not been received, the protocol processing unit 36 proceeds to S44.

In S44, the clock control unit 34 of the communication path in which the link failure has occurred determines whether or not the clock 45 corresponding to the other communication path is synchronized with the time of the time server 11. Specifically, for example, when a link failure occurs in the first communication path, the clock control unit 34 corresponding to the first communication path synchronizes the clock 45 corresponding to the second communication path with the time of the time server 11. Check if it is done. For example, if the clock control unit 44 sets the synchronization flag when synchronizing with the time of the time server 11, the clock control unit 34 refers to the synchronization flag of the clock control unit 44 to clock. You may check the presence or absence of synchronization of 45. The method of confirming synchronization is not limited to this example.
If the clock control unit 34 can confirm that the clock 45 corresponding to the other communication path is synchronized with the time of the time server 11, the process proceeds to S45, and if not, the process proceeds to S48.

In S45, the clock control unit 34 reads the time information held by the clock control unit 44 corresponding to another communication path. Alternatively, the clock control unit 34 may read the correction amount calculated by the protocol processing unit 46 corresponding to another communication path. The clock control unit 34 proceeds to S46 when the reading of the time information is completed.
In S46, the clock control unit 34 calculates the correction amount of the clock 35 from the time information read from the clock control unit 44 corresponding to the other communication path. When the clock control unit 34 reads the already calculated correction amount in S45, the clock control unit 34 uses the read correction amount as it is as the calculated correction amount. The clock control unit 34 proceeds to S47 when the correction amount is calculated.

In S47, the clock control unit 34 corrects the time of the clock 35 based on the calculated correction amount. When the correction of the clock 35 is completed, the clock control unit 34 proceeds to S53. The protocol processing unit 36 does not change the transmission interval of the PTP packet before and after the occurrence of the link failure, and continues to transmit the PTP packet at a constant interval.
When the correction of the clock 35 is completed, the protocol processing unit 36 operating as the PTP master can transmit the PTP packet using the clock 35 synchronized with the time server 11. As a result, the PTP synchronization process can be continued for the sensor system 20 after the location where the link failure has occurred (downstream side), and the PTP synchronization of the photographing system 10 can be maintained.
In the present embodiment, the clock corresponding to the communication path in which the link failure has occurred is synchronized with the clock in which the link failure has not occurred (synchronized with the time server 11), so that the downstream side connected in the daisy chain is synchronized. Time synchronization with the device can be maintained.

In S48, the clock control unit 34 determines which time information of the clock 35 or the clock 45 is used to execute the synchronization process, and then proceeds to S49.
In the situation advanced to S48, neither the clock 35 corresponding to the first communication path of the sensor system 20 nor the clock 45 corresponding to the second communication path are PTP synchronized with the time server 11. Therefore, which clock time information is used for synchronization processing with the clock of the sensor system on the downstream side is determined.
Specifically, when the clock 35 corresponding to the first communication path and the clock 45 corresponding to the second communication path are not PTP-synchronized, the other clock is used as the clock corresponding to the predetermined communication path. Synchronize. Alternatively, the time information held by the clock control units (34, 44) may be synchronized with the clock corresponding to the communication path that has been synchronized with the time server 11 until the latest.

In S49, the clock control unit 34 determines whether or not the time of the clock 35 is synchronized with the time of the clock 45 based on the determination in S48. If the time of the clock 35 is synchronized with the time of the clock 45, the process proceeds to S50, otherwise the process proceeds to S53. When the process proceeds from S49 to S53, the clock control unit 44 that executes the same process separately synchronizes the time of the clock 45 with the time of the clock 35.
In S50, the clock control unit 34 reads the time information of the clock 45 held by the clock control unit 44 corresponding to another communication path. Alternatively, the clock control unit 34 may read the already calculated correction amount from the clock control unit 44 of another communication path. When the clock control unit 34 reads the time information, it proceeds to S51.

In S51, the clock control unit 34 calculates the correction amount from the time information read from the clock control unit 44 corresponding to the other communication path. Alternatively, when the clock control unit 34 has read the already calculated correction amount in S50, the correction amount is used as it is as the calculated correction amount. The clock control unit 34 proceeds to S52 when the correction amount is calculated.

In S52, the clock control unit 34 corrects the clock 35 with the calculated correction amount.
When the correction of the clock 35 is completed, the clock control unit 34 proceeds to S53. By completing the correction of the clock 35, the protocol processing unit 36 transmits a PTP packet to the sensor system 20 after the location where the link failure has occurred by using the clock synchronized between the communication paths. The PTP synchronization process can be continued. The protocol processing unit 36 does not change the transmission interval of the PTP packet before and after the occurrence of the link failure, and continues to transmit the PTP packet at a constant interval.

Here, about the reason why synchronization is maintained by using either clock even if neither the clock corresponding to the first communication path nor the clock corresponding to the second communication path can be synchronized with the time of the time server 11. explain.
As described above, each sensor system 20 has a function of storing image data in a storage device in correspondence with time information. Therefore, even if PTP synchronization with the time server 11 is not possible, if the time of each sensor system 20 is synchronized, the time information of the image data acquired by each sensor system 20 can be synchronized. Become a state. As a result, for example, in the image processing server 14, it becomes possible to easily perform a composition process or the like using synchronized image data, and to generate high-quality virtual viewpoint contents or the like.

In S53, the protocol processing unit 36 determines whether or not to continue the operation of the sensor system 20. Specifically, for example, when the control terminal 13 stops the synchronization processing of the entire sensor systems 20a to 20z, or when only the specified one of the sensor systems 20a to 20z stops the synchronization processing. It is determined that the operation of the sensor system 20 is not continued.
If the operation of the sensor system 20 is to be continued, the protocol processing unit 36 returns to S42 to continue the processing, and if not, the processing is terminated.

As described above, in the present embodiment, even if a link failure occurs in any of the plurality of communication paths, time synchronization with respect to the downstream device connected in the daisy chain can be maintained. This makes it possible to improve the certainty of time synchronization between a plurality of devices connected in a daisy chain.

(Modification example)
In the above-described embodiment, a configuration including one clock 35 (45) for each communication path has been described as an example, but the number and arrangement of clocks are not limited to this example. Hereinafter, in other configuration examples, synchronization processing when a failure occurs will be mainly described.
A configuration having two clocks for each communication path, for example, a configuration having a corresponding clock for each communication port (31a, 38b, 41a, 48a) may be used. In this case, the time synchronization processing between the clocks corresponding to the communication ports (31a, 38a) of the first communication path and the clocks corresponding to the communication ports (41a, 48a) of the second communication path is executed.

For example, when a clock failure corresponding to the communication port 31a occurs, the clock corresponding to the communication port 38a cannot be synchronized with the time of the GM device and another BC device operating the PTP master. In such a case, the clock corresponding to the communication port 38a is synchronized with one of the clocks of the second communication path (the clock corresponding to the communication port 41a and the clock corresponding to the communication port 48a). As a result, the clock corresponding to the communication port 38a can be synchronized with the time of the GM device, the PTP packet can be transmitted to the subsequent device as the master device (Mst), and the synchronization can be maintained throughout the system. can.

Further, when a link failure or the like occurs in the first communication path, the clocks (31a, 38a) cannot be synchronized with the GM device and the BC device operating as the PTP master.
In this case, the clock (31a) may be synchronized with either the clock (41a, 48a) of the second communication path, or the clock (38a) may be synchronized with either the clock (41a, 48a) of the second communication path. You may.
Further, even when the clock (41a) fails or a link failure occurs in the second communication path, the synchronization of the entire system can be maintained by synchronizing the clocks in the same manner.

Further, a configuration in which a clock is shared between the first communication path and the second communication path, for example, the first communication port 31a and the second communication port 41a have one common clock, and the third communication port 38a and the fourth communication port are provided. Each of the 48a may be provided with a clock.
In this case, either the PTP packet received via the first communication path or the second communication path is selected, and the synchronization process of the common clock is executed. Further, a time synchronization process is executed between the common clock and the clock corresponding to the third communication port 38a, or between the common clock and the clock corresponding to the fourth communication port 48a.
When a link failure occurs in both the first communication path and the second communication path, the common clock, the clock corresponding to the third communication port 38a, and the clock corresponding to the fourth communication port 48a are the GM device and the PTP, respectively. It becomes impossible to synchronize with the BC device that operates as a master. In this case, by synchronizing the clock corresponding to the third communication port 38a and the clock corresponding to the communication port 48a to the common clock or one of the clocks, the synchronization of the entire system can be maintained. ..

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof. Further, the present invention can be implemented as, for example, a system, an apparatus, a method, a program, a recording medium (storage medium), or the like.

In addition, the present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or an apparatus via a network or a storage medium, and one or more processors in the computer of the system or the apparatus provide the program. It can also be realized by the process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. ASIC is an abbreviation for Application Specific Integrated Circuit.

10 ... Shooting system, 11 ... Time server, 12 ... Hub, 13 ... Control terminal, 14 ... Image processing server, 15 ... User terminal, 17 ... GNSS, 20 ... Sensor system, 21 ... Camera, 22 ... Camera adapter, 23 ... Camera control unit, 24 ... image processing unit, 25 ... network control unit, 33a, 43a ... RTP packet detection unit, 34a, 44a ... clock control unit, 35a, 45a ... clock, 36a, 46a ... protocol processing unit

Claims (8)

A first correction means for synchronizing the first time information with the synchronization information received via the first communication path, and
A second correction means for synchronizing the second time information with the synchronization information received via the second communication path, and
When the synchronization information via the first communication path or the synchronization information via the second communication path is not received, the time information corresponding to the communication path from which the synchronization information is not received is the other time information. A synchronization method that synchronizes with time information,
A first transmission means for transmitting synchronization information based on the first time information to a subsequent device via the first communication path, and
A second transmission means for transmitting synchronization information based on the second time information to a subsequent device via the second communication path, and a second transmission means.
A communication device characterized by being provided with. The first transmission means and the second transmission means transmit the synchronization information at regular intervals.
The communication device according to claim 1. After starting the synchronization, the synchronization means ends the synchronization when the synchronization information is received via the communication path for which the synchronization information was not received.
The communication device according to claim 1 or 2, wherein the communication device is characterized in that. The synchronization means is
A calculation means for calculating the correction amount in the same period is provided.
When the correction amount calculated by the calculation means is larger than a predetermined threshold value, the synchronization is performed step by step.
The communication device according to any one of claims 1 to 3, wherein the communication device is characterized by the above. The synchronization information is a PTP (Precision Time Protocol) synchronization packet.
The communication device according to any one of claims 1 to 4, wherein the communication device is characterized in that. The synchronization means is
When neither the synchronization information via the first communication path nor the synchronization information via the second communication path is received, either the first time information or the second time information is used. Equipped with a decision-making means to decide whether to use as a standard
Synchronize the other time information with the time information determined by the determination means.
The communication device according to any one of claims 1 to 5, wherein the communication device is characterized in that. A step of synchronizing the first time information with the synchronization information received via the first communication path, and
A step of synchronizing the second time information with the synchronization information received via the second communication path, and
When the synchronization information via the first communication path or the synchronization information via the second communication path is not received, the time information corresponding to the communication path from which the synchronization information is not received is the other time information. Steps to synchronize with time information and
A step of transmitting synchronization information based on the first time information to a subsequent device via the first communication path, and a step of transmitting the synchronization information to a subsequent device.
A step of transmitting synchronization information based on the second time information to a subsequent device via the second communication path, and a step of transmitting the synchronization information to a subsequent device.
A communication method characterized by having. A program for making a computer function as each means of the communication device according to any one of claims 1 to 6.

Images