JP2024055333A - IMAGING APPARATUS, CONTROL METHOD FOR IMAGING APPARATUS, PROGRAM, INFORMATION PROCESSING APPARATUS, AND IMAGING SYSTEM - Google Patents

Abstract

[Problem] To provide an imaging device that calculates and outputs a differential time from a delay time so that a synthesized image does not look unnatural from a still image. [Solution] The imaging device is characterized by having an imaging section that performs image processing on an image captured in accordance with a signal output from a synchronization signal transmission device that transmits a periodic signal to generate an image, a pan-tilt section that can change the angle of view of the imaging section, an output section that outputs the image generated by the imaging section, a packet generation section that obtains status information that is information on the status of the pan-tilt section when the signal output from the synchronization signal transmission device is detected and generates a packet including the status information, and a time addition section that calculates a differential time that is the difference between a first time that is the time when the status information was obtained and a second time that is the time when the packet generation section outputs the packet, and adds the differential time to the packet. [Selected Figure] Figure 2

Description

The present invention relates to an imaging device, a control method for an imaging device, a program, an information processing device, and an imaging system.

In recent years, there has been a demand for content that combines images captured by a camera with background images of a virtual space. In particular, content in which images from a pan-tilt camera are sent to an information processing device such as a computer and combined with images of a virtual space generated by the information processing device is in demand for use in video distribution and virtual conferences. To combine such pan-tilt cameras with virtual spaces, there is a protocol that transmits information about the camera's attitude, including the pitch angle and yaw angle of the pan-tilt camera, via a network such as ETHERNET (registered trademark). This protocol allows an imaging device to act as a camera in the world coordinate system of the virtual space created by the information processing device, and an example of this is the FreeD protocol.

On the other hand, there is a known technology that uses a phase synchronization signal for the camera's HDMI (registered trademark) video to synchronize the timing of the video and correct errors in video generation between cameras, assuming the linking of multiple cameras. Given this technical background, in an environment where multiple cameras are linked, there is a demand for a combination of video synchronization using a synchronization signal and video synchronization in a virtual space such as the FreeD protocol.

The purpose of such corrections using phase synchronization signals is to correct misalignment between frames, but because the FreeD protocol is distributed over a network, it requires a latency greater than one frame, and the amount of delay is greater than with video signals. One technique for synchronizing a video signal with mechanical drive information such as FreeD is the technique shown in Patent Document 1, which synchronizes a camera with a pan head with video, but this only synchronizes time within one frame, such as the start and end timing of pan-tilt drive.

In addition, in Patent Document 2, synchronization is performed based on a hard line between different devices such as a lens and a camera, but synchronization is not performed using different communication systems such as a video signal and a network.

JP 2019-87880 A JP 2017-175380 A

However, Patent Documents 1 and 2 do not synchronize the synchronized video signal within a frame with delays of different scales, such as delays in information transmitted over a network for pan/tilt information.

Even if the time difference within a frame is synchronized, the time that the pan/tilt information is provided is delayed, so differences arise when the 3D graphics generated using the pan/tilt information are combined with the video information. For example, if an image when the pan/tilt has stopped is combined with a 3D graphic based on the pan/tilt information just before it has stopped while the pan/tilt is still operating, the image generated from the 3D graphic (3D graphic image) is generated based on the information while the pan/tilt is operating. As a result, the user feels uncomfortable when the image is combined with the stopped image.

The present invention aims to provide an imaging device that calculates and outputs a difference time from the delay time so that the synthesized image does not look unnatural compared to a still image.

In order to achieve the above object, an imaging device according to one aspect of the present invention is characterized by having an imaging unit that performs image processing on an image captured in accordance with a signal output by a synchronization signal transmission device that transmits a periodic signal, and generates a video, a pan-tilt unit that can change the angle of view of the imaging unit, an output unit that outputs the video generated by the imaging unit, a packet generation unit that acquires status information that is information on the status of the pan-tilt unit when the signal output by the synchronization signal transmission device is detected, and generates a packet including the status information, and a time addition unit that calculates a differential time that is the difference between a first time that is the time when the status information was acquired and a second time that is the time when the packet generation unit outputs the packet, and adds the differential time to the packet.

The present invention provides an imaging device that calculates and outputs a difference time from the delay time so that the synthesized image does not look unnatural compared to a still image.

FIG. 1 is a diagram showing a system configuration according to a first embodiment. FIG. 2 is a diagram showing functional blocks of a system according to the first embodiment. 10 is a diagram showing a time correlation of control in a case where there is no pan/tilt information and image difference information in the first embodiment. FIG. 5 is a diagram showing a time correlation of control in the information processing device based on pan/tilt information and image difference information in the first embodiment. FIG. FIG. 4 is a diagram showing a processing flow in an image capturing processing unit 201 according to the first embodiment. FIG. 4 is a diagram showing a processing flow in a communication processing unit 202 according to the first embodiment. FIG. 4 is a diagram showing a processing flow in the information processing device 102 according to the first embodiment. FIG. 13 is a diagram showing a system configuration in which there is a difference in differential time between imaging devices according to a second embodiment; FIG. 11 is a diagram showing a system configuration of an imaging device and an information processing device in which difference times are synchronized according to a second embodiment. FIG. 11 is a diagram showing functional blocks of a system according to a second embodiment. 13A and 13B are diagrams illustrating a case where the first imaging device according to the second embodiment operates based on different difference times to perform synchronization processing. 13A and 13B are diagrams illustrating a case where the second imaging device according to the second embodiment operates based on different difference times to perform synchronization processing. 13A and 13B are diagrams illustrating a case where the third imaging device according to the second embodiment operates based on different difference times to perform synchronization processing. 13 is a diagram illustrating a time correlation of control in an information processing apparatus when there is a difference in differential time according to the second embodiment. FIG. 13 is a diagram showing a time relationship in a case where the first imaging device outputs packets and video with a delay corresponding to the largest time difference according to the second embodiment. FIG. 13 is a diagram showing a time relationship in a case where the second imaging device outputs packets and video with a delay corresponding to the largest time difference according to the second embodiment. FIG. 13 is a diagram showing a time relationship when the third imaging device outputs packets and video with a delay corresponding to the largest time difference according to the second embodiment. FIG. FIG. 11 is a diagram showing a processing flow of an imaging processing unit 201 according to the second embodiment. FIG. 11 is a diagram showing a processing flow of a communication processing unit 202 according to the second embodiment.

Below, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. However, the embodiments of the present invention are not limited to the following embodiments. The same or equivalent components, parts, and processes shown in each drawing will be given the same reference numerals, and duplicated explanations will be omitted as appropriate. In addition, some parts that are not important for the explanation will be omitted in each drawing.

<Embodiment 1>
1 is a diagram showing a system configuration in this embodiment. The system (imaging system) in this embodiment is configured to include a first imaging device 100, a second imaging device 105, a synchronization signal transmission device 106, an information processing device 102, and a display 104. The first imaging device 100, the second imaging device 105, and the information processing device 102 are connected to each other via a network 103. The network 103 is realized by a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark). The network 103 may be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), etc.

The first imaging device 100 is a device that captures (takes) an image. The first imaging device 100 is equipped with a camera platform, and includes an imaging lens and an imaging sensor (solid-state imaging element) such as a CCD or CMOS, and functions as an imaging means that can change the imaging range by panning and tilting. The first imaging device 100 is connected to an external device such as an information processing device 102 via a line and a network 103 so as to be able to communicate with the external device.

The first imaging device 100 can capture images while changing the imaging range by panning and tilting using a pan-tilt unit described later. When the first imaging device 100 receives a synchronization signal from a synchronization signal transmitter 106, the imaging sensor captures an image, processes the image, and generates a video. The generated video is converted into a video signal and output to the information processing device 102 via a line. The control value (pan-tilt information) of the pan-tilt unit 203 when the video is output is transmitted (sent) to an external device such as the information processing device 102 via the network 103. The line is, for example, a cable that complies with a communication standard such as HDMI (registered trademark) (High Definition Multimedia Interface).

The first imaging device 100 includes a central processing unit (CPU) (not shown) as a central processing unit, and memories such as a RAM (andom access memory) and a ROM (read-only memory). The CPU is connected to each component of the imaging device via a line, and performs overall control of the operation of each component of the entire imaging device, such as adjusting the operation of each component, according to a computer program stored in the ROM (read-only memory).

The information processing device 102 is, for example, a client device such as a personal computer (PC) in which a program for implementing the processing functions described below is installed. Although not shown, the information processing device 102 includes one or more processors such as a CPU or MPU, memory such as a RAM or ROM, a HDD, drivers for each I/F, etc.

The CPU is a central processing unit that performs calculations and logical judgments for various processes and generally controls each component of the information processing device 102. The ROM is a program memory that stores computer programs for control by the CPU. The RAM is used as a temporary storage area such as the CPU's main memory and work area. Note that the program memory may be realized by loading a program into the RAM from an external storage device connected to the information processing device 102.

The HDD is a hard disk for storing electronic data and programs. An external storage device may also be used to fulfill a similar role. Here, the external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories. The external storage device may also be a server device connected via a network.

The I/F unit is an interface for connecting to a removable device, and includes, for example, a power source and an attachment mechanism such as a removable socket for attaching and detaching the removable device. The information processing device 102 performs data communication with the removable device via the I/F unit. Although not shown, this system also has an operation unit connected to the information processing device 102 via a line. The operation unit has a keyboard and a mouse, and accepts various operations by the user. Note that the operation unit is an external device connected to the information processing device 102 via a line, but the operation unit may be provided in the same housing as the information processing device 102.

The display (display unit) 104 is configured with an LCD (Liquid Crystal Display) or the like, and functions as a display means for displaying images and the like synthesized by the information processing device 102. The display 104 is communicably connected to the information processing device 102 via a line, and in the first embodiment, the display 104 is connected to the information processing device 102 via a display cable that complies with a communication standard such as HDMI (registered trademark) as the line. In the first embodiment, the display 104 is an external device connected to the information processing device 102 via a line, but the display 104 may be provided in the same housing as the information processing device 102.

The second imaging device 105 is a device that captures (takes) an image. The second imaging device 105 includes a camera platform, an imaging lens, and an imaging sensor (solid-state imaging element) such as a CCD or CMOS, and functions as an imaging means that can change the imaging range by panning and tilting. The second imaging device 105 is communicably connected to an external device such as an information processing device 102 via a line and a network 103. The second imaging device 105 can capture an image while changing the imaging range by panning and tilting using a pan-tilt unit described later. The second imaging device 105 captures an image with the imaging sensor at the timing when it receives a synchronization signal from a synchronization signal transmission device 106, performs image processing, and generates an image. The generated image is converted into a video signal and output to the information processing device 102 via the line. The control value (pan-tilt information) of the pan-tilt unit 203 when the image is output is also transmitted to an external device such as the information processing device 102 via the network 103.

In the system according to the first embodiment, the first imaging device 100 and the second imaging device 105 are described as being one each, but there may be two or more. That is, two or more first imaging devices 100 and second imaging devices 105 can be connected to the information processing device 102 via a line or network 103 and can transmit video signals to the information processing device 102.

The synchronization signal transmission device 106 outputs a signal (synchronization signal) to the first imaging device 100 and the second imaging device 105 at the same frequency as the frequency at which the image 310 is captured, and notifies the timing at which the captured images of the first imaging device 100 and the second imaging device 105 are created. Specifically, the synchronization signal transmission device 106 transmits the transmission timing of the image signals of the second imaging device 105, the first imaging device 100, and the second imaging device 105 as an electrical signal. The first imaging device 100 and the second imaging device 105 perform imaging processing at the timing of receiving the signal output from the synchronization signal transmission device 106, and output the image 310 captured at the same timing within the width of the frame time. In other words, the synchronization signal transmission device 106 can make the timing of the imaging processing of the multiple imaging devices the same.

FIG. 2 is a diagram showing the functional blocks of the system according to the first embodiment. Specifically, an example of the functional blocks of the first imaging device 100 and the information processing device 102 is shown. Note that each function of the imaging device and the information processing device 102 shown in FIG. 2 is realized as follows. That is, the CPU of the first imaging device 100 or the second imaging device 105 executes a computer program stored in a ROM provided in each imaging device and the information processing device 102. Here, although the above description is of the CPU of the first imaging device 100 or the second imaging device 105, the same applies when there are three or more imaging devices. Note that each function on the imaging device side and each function on the information processing device side may be realized by the CPU in each device executing a computer program stored in a ROM provided in each device.

The first imaging device 100 in the first embodiment has an imaging processing unit (imaging control unit) 201 and a communication processing unit (communication control unit) 202. In the first embodiment, the imaging processing unit 201 and the communication processing unit 202 each have a different CPU, and the imaging processing unit 201 and the communication processing unit 202 communicate with each other via in-flight communication. Note that the imaging processing unit 201 and the communication processing unit 202 may be processed by the same CPU, or may be processed by different cores, or may be divided into software modules within the same CPU.

First, the configuration of the imaging processing unit 201 will be described. The imaging processing unit 201 is configured to include a pan/tilt unit 203, an information storage unit (pan/tilt information storage unit) 204, a time providing unit 205, an imaging unit 206, and a video signal output unit 207.

The pan-tilt unit 203 drives the imaging unit 206 in the pan direction (left and right direction) and tilt direction (up and down direction) by motor drive, and functions as a mechanism (pan-tilt mechanism) that can change the angle of view when the first imaging device 100 captures images.

The information holding unit 204 detects the timing of the electrical signal from the synchronization signal transmitter 106 through notification from the imaging unit 206, acquires pan-tilt information from the pan-tilt unit 203, and acquires the pan-tilt information acquisition time (first time) from the time providing unit 205 on the camera side. Here, the pan-tilt information is status information of the pan-tilt unit 203 including information on the pan-tilt information that is the current angle of view of the pan-tilt unit 203 (or the imaging unit 206). The pan-tilt information acquisition time is the time at the timing when the pan-tilt information is acquired. The information holding unit 204 holds the pan-tilt information and the pan-tilt information acquisition time. The information holding unit 204 also transmits the acquired pan-tilt information and pan-tilt information acquisition time to the communication processing unit 202 through in-device communication.

The time providing unit 205 is a time providing unit on the camera (imaging processing unit 201) side that is capable of providing the same reference time (synchronized time) by synchronizing with the time providing unit 209 on the network (communication processing unit 202) side, which will be described later. The time providing unit 205 transmits the pan/tilt information acquisition time to the information holding unit 204 as described above.

The imaging unit 206 processes the sensor information obtained from the imaging element and outputs it as an image. The imaging unit 206 also receives an electrical signal (synchronization signal) from the synchronization signal transmission device 106 and starts imaging in accordance with the reception timing. The imaging unit 206 performs imaging processing (image processing) at the timing output from the synchronization signal transmission device 106, and generates an image 310 captured at the same timing within the width of the frame time. Also, by receiving a similar electrical signal from the second imaging device 105 in addition to the first imaging device 100 and performing imaging in accordance with the signal, it is possible to capture images at the same timing as the second imaging device 105 and generate an image 310 that is an image based on each captured image.

The video signal output unit 207 converts the video 310 generated by the imaging unit 206 into a video signal, and outputs the video 310 converted into a video signal to the information processing device 102. The video signal output unit 207 outputs the video 310 converted into a video signal to the information processing device 102 in accordance with (in synchronization with) the signal output by the synchronization signal transmission device 106.

Next, the configuration of the communication processing unit 202 will be described. The communication processing unit 202 is configured to include a packet generation unit 208, a time provision unit 209, a time addition unit (pan/tilt time addition unit) 210, and a communication unit 211.

After receiving the pan-tilt information from the information storage unit 204, the packet generation unit 208 converts the pan-tilt information (shapes the data) into a packet format that can be received by the information processing device 102 via the network 103 to generate a packet. At this time, the pan-tilt information acquisition time added to the pan-tilt information is not included in the packet, i.e., it is sent to the time addition unit 210 without being converted into a packet format.

The time providing unit 209 is a time providing unit on the network (communication processing unit 202) side that is capable of providing the same reference time by synchronizing with the time providing unit 205 on the camera side. In the first embodiment, the time providing unit 209 on the network side and the time providing unit 205 on the camera side each have a mechanism for updating the passage of time. For example, the time providing unit 205 holds a shared memory and periodically checks the respective times, and if a discrepancy occurs, it synchronizes with the time providing unit 209 to achieve time synchronization using a mechanism that suppresses the time discrepancy.

The time addition unit 210 acquires the packet sending time (second time) from the time provision unit 209 immediately before the packet is output from the communication unit 211. Then, based on the packet sending time and the pan/tilt information acquisition time transmitted from the packet generation unit 208, the time addition unit 210 calculates the difference between the packet sending time and the pan/tilt information acquisition time as the differential time 401, and adds the differential time 401 to the pan/tilt information (adds it to the packet). After adding the differential time 401 to the packet, the time addition unit 210 transmits the packet of pan/tilt information to which the differential time has been added to the information processing device 102 via the communication unit 211. In the first embodiment, the pan/tilt information is formed into a packet in compliance with the FreeD protocol and transmitted to the information processing device 102.

The communication unit 211 is a communication unit on the imaging device side, and transmits and receives various data to and from the second imaging device 105 and the information processing device 102 via the network 103. For example, the communication unit 211 transmits a packet of pan-tilt information to which the difference time 410 has been added to the information processing device 102 as data transmission.

Next, the second imaging device 105 will be described. The second imaging device 105 has the same configuration as the first imaging device 100, i.e., the same software and hardware configuration. It is also assumed that the difference time 401 acquired by the second imaging device 105 is the same as that of the first imaging device 100 or is a time within one frame. The second imaging device 105 transmits the image 310 converted into a video signal and a packet of pan-tilt information to which the difference time 401 has been added, to the information processing device 102, in the same way as the first imaging device 100 described above.

Next, the information processing device 102 will be described. The information processing device 102 acquires video signals from the first imaging device 100 and the second imaging device 105 via a line. Furthermore, it acquires pan-tilt information with added differential time from the first imaging device 100 and the second imaging device 105 via a network 103. The information processing device 102 in the first embodiment generates three-dimensional graphics and generates content that combines the video 310 sent from the first imaging device 100 or the second imaging device 105 with the three-dimensional graphics.

When generating a three-dimensional graphic image, rendering is performed on the assumption that an imaging device exists in the virtual space based on the pan-tilt information received from the network 103, in order to composite it with the image 310 as an imaging device in the three-dimensional graphic coordinate system. At this time, rendering is performed by calculating the angle of view so that the angle of view is the same. Then, the rendered image (three-dimensional graphic image) is composited with the received image 310 to generate an image that combines the image 310 with the angle of view generated by the three-dimensional graphics.

The information processing device 102 in the first embodiment is configured to include a communication unit 212, a pan-tilt information drawing unit 213, a three-dimensional graphics generating unit 214, an image storage unit 215, and a drawing synthesis unit 216.

The communication unit 212 is a communication unit on the information processing device 102 side, and transmits and receives various data to and from the first imaging device 100 and the second imaging device 105 via the network 103. When the communication unit 212 receives pan-tilt information converted into packets from the first imaging device 100 and the second imaging device 105, it transmits the received pan-tilt information to the pan-tilt information drawing unit 213.

The pan/tilt information drawing unit 213 changes the drawing of the three-dimensional graphic data generated by the three-dimensional graphic generation unit 214 described below, and generates a three-dimensional graphic image. Specifically, it determines the camera angle of view for the three-dimensional graphic based on the pan/tilt information of the first imaging device 100 and the second imaging device 105 transmitted from the communication unit 212. Then, it performs rendering on the three-dimensional graphic data, and generates a three-dimensional graphic image.

The three-dimensional graphics generation unit 214 generates three-dimensional graphics data to be combined with the video 310 acquired from the first imaging device 100 and the second imaging device 105. After the three-dimensional graphics data is generated, it is sent to the pan-tilt information drawing unit 213. Note that in the first embodiment, the three-dimensional graphics generation unit 214 does not perform rendering, but the pan-tilt information drawing unit 213 performs rendering as described above. Also, the pan-tilt information drawing unit 213 may calculate the camera angle of view in the three-dimensional graphics based on the pan-tilt information, and the three-dimensional graphics generation unit 214 may perform drawing.

The image holding unit 215 holds the image 310 transmitted from the first imaging device 100 and the second imaging device 105 for a certain period of time, and provides the image 310 for a specified period of time to the drawing synthesis unit 216.

The drawing synthesis unit 216 synthesizes the three-dimensional graphic image generated by the pan/tilt information drawing unit 213 and rendered at the determined angle of view with the image 310 stored in the image storage unit 215 to generate (create) a synthetic image. When synthesizing, it receives the difference time 401 corresponding to the pan/tilt information used for drawing from the pan/tilt information drawing unit 213, and obtains the image 310 before the difference time 401 from the image storage unit 215 and synthesizes it. By correcting the difference between the pan/tilt information and the image in the difference time 401 when synthesizing, it is possible to obtain and synthesize the image 310 during the pan/tilt operation corresponding to the pan/tilt information. This makes it possible to correct the difference between the generated three-dimensional graphic image and the image 310, and it is possible to obtain a synthetic image that does not look unnatural even when compared to a stationary image.

Figure 3 is a diagram showing the time correlation of control when there is no information on the difference between pan/tilt information and the image. In Figure 3, a difference time 401 between the pan/tilt information and the image occurs, and the state in which a difference occurs between the pan/tilt state of the image and the pan/tilt information and the image 310 over time is shown. Figure 4 is a diagram showing the time correlation of control in an information processing device based on the pan/tilt information and the information on the difference between the image. In Figure 4, the state in which the image and the pan/tilt information are synchronized and the difference is corrected over time by transmitting the difference time 401 is shown. Below, Figure 3 will first be described.

The horizontal axis in FIG. 3 indicates the time axis, and shows the data flow in the first imaging device 100 and the information processing device 102. n indicates an arbitrary frame number, and in FIG. 3, it indicates that images 310 up to n+12 have been output. The image processing time 301 indicated by the image signal path 308 indicates the time until the image 310 processed by the imaging unit 206 is output from the image signal output unit 207. The pan/tilt information path 309 indicates the time until the pan/tilt information from n to n+11 is generated, and the pan/tilt information for three frames is generated in parallel. The time until the pan/tilt information is generated is indicated by the sum of the pan/tilt acquisition time 302, the in-flight communication time 303, the packet generation time 304, and the packet output time 305.

Pan/tilt acquisition time 302 indicates the time until pan/tilt information is generated and the time until the information storage unit 204 acquires information from the pan/tilt unit 203, generates pan/tilt information, and transmits it to the packet generation unit 208. In-flight communication time 303 indicates the time until the pan/tilt information is communicated from the information storage unit 204 of the imaging processing unit 201 to the packet generation unit 208 of the communication processing unit 202. Packet generation time 304 indicates the time until the packet generation unit 208 generates a packet from the pan/tilt information and the time addition unit 210 adds the difference time 401. Packet output time 305 indicates the time until the packet to which the difference time 401 is added is output.

In FIG. 3, when a packet is generated using a conventional method that does not add differential time 401, the in-flight communication time 303 varies depending on the type and frequency of in-flight communication, such as SPI or PCIe. Note that in the first embodiment, it is assumed that communication takes time because it is assumed that communication is performed using a slow communication format. Also, the packet generation time 304 and packet output time 305 vary depending on the steady-state processing load and priority of the communication processing unit 202, but in the first embodiment, it is assumed that generation and output take time.

In the information processing device 102, it shows the time from receiving the image 310 and pan/tilt information to synthesizing them. Image 310 shows the time from when the information holding unit 204 holds the image 310 output from the video signal output unit 207. Time 311 in the pan/tilt information drawing unit 213 shows the time from when the information processing device 102 acquires the pan/tilt information to when the pan/tilt information drawing unit 213 performs drawing using 3D graphics. Time 312 in the drawing synthesis unit 216 shows the time when the drawing synthesis unit 216 synthesizes the 3D graphic image drawn by the pan/tilt information drawing unit 213 with the image 310 held in the image holding unit 215, and the synthesized frame numbers from n to n+11.

Here, assume that the difference time 401 is not transmitted and a composite image is generated from the pan/tilt information and image 310 acquired immediately before. At this time, if the image drawn from the currently acquired information and the image held in the image holding unit 215 are notified without being calculated, it becomes unclear which image should be combined with the image 310 held in the image holding unit 215. For example, if the image is combined based on the latest image 310 received and the pan/tilt information, the pan/tilt information is received with a shift of two frames, so the generated image will have a difference of two frames. In such an image, if a sudden change occurs in the image, such as immediately before or after the pan/tilt stops, a difference will occur between the image drawn by the pan/tilt information drawing unit 213 and the image 310 held in the image holding unit 215. Therefore, in order to prevent such a difference from occurring, it is necessary to measure the difference time 401 and shift the image based on the three-dimensional graphic image drawn by the pan/tilt information drawing unit 213 of the information processing device 102 and the image 310 held in the image holding unit 215.

Figure 4 shows the time relationship when a differential time 401 is added to the time adding unit 210 based on the method proposed in the first embodiment and a packet is output. The video signal path 308, video 310, and time 311 of the pan/tilt information drawing unit are the same as those shown in Figure 3, so the explanation is omitted.

In addition, in FIG. 4, the difference time 401 is added to the packet and sent via the pan/tilt information path 309. Therefore, at time 312 in the drawing synthesis unit 216, the drawing synthesis unit 216 acquires the difference time 401, and it becomes possible to determine which frame image of the image 310 stored in the image storage unit 215 corresponds to the pan/tilt information.

In the first embodiment, by synthesizing the image 310 delayed by two frames with the image rendered based on the pan-tilt information, it is possible to synthesize the image 310 with the corresponding pan-tilt and the three-dimensional graphic image based on the pan-tilt information. The difference time 401 does not have to be an exact time as long as the time difference can be calculated using something such as a frame number.

However, in the first embodiment, by sending the time, it is possible to handle cases where the video signal is transmitted in various patterns. For example, in the imaging unit 206, the video is synchronized and output by the synchronization signal transmission device 106, but even in cases where the video signal output unit 207 thins out the frequency and outputs it, it is possible to synchronize the video and pan/tilt information by sending the time. Note that in the first embodiment, since the second imaging device 105 also has the same hardware and software configuration, the difference time 401 is almost equal to the difference time 401 of the first imaging device 100 and does not deviate significantly.

Synchronization between imaging devices with approximately the same differential time 401 can be achieved by delaying the images for both imaging devices in the same way and synthesizing them, so that the pan/tilt information and images of either imaging device can be synchronized to create a composite image. In addition, in the first embodiment, it is assumed that the differential time 401 is a time greater than the time of one frame (a predetermined time or more). If it is a time less than the time of one frame, it is not necessarily necessary to delay the images by the differential time 401 to synthesize them. Therefore, in the first imaging device 100 of the first embodiment, if the differential time 401 is less than one frame, it is preferable not to add the differential time 401 to a packet and send it to the information processing device 102. In addition, in the information processing device 102 of the first embodiment, if the differential time 401 is not sent from the imaging device, or if the differential time 401 is sent but is less than one frame, the above synchronization process is not performed.

Figures 5, 6, and 7 show the flow of synchronizing the pan-tilt information and the image 310 based on the difference time 410 in embodiment 1.

Figure 5 is a diagram showing a processing flow in the imaging processing unit 201 according to the first embodiment. Figure 6 is a diagram showing a processing flow in the communication processing unit 202 according to the first embodiment. Figure 7 is a diagram showing a processing flow in the information processing device 102 according to the first embodiment. Note that each process (each operation) in Figures 5 and 6 is controlled by the CPU in the first imaging device 100 executing a computer program stored in the ROM of each of the first imaging device 100 and the information processing device 102. Also, each process (each operation) in Figure 7 is controlled by the CPU in the information processing device 102 executing a computer program stored in the ROM. Also, by adding an S to the beginning of each process (step), the notation of the process (step) is omitted. Also, the imaging device shown in the following process will be described as the first imaging device 100, but the same applies to the second imaging device 105.

The processing flow of the imaging processing unit 201 shown in FIG. 5 describes the process up to outputting the image 310 converted into a video signal and outputting the pan/tilt information to the packet generating unit 208. The entire process shown in FIG. 5 is performed periodically.

First, in S1001, the time providing unit 205 on the camera side performs time synchronization with the communication processing unit 202 every cycle (predetermined cycle). Next, in S1002, it is determined whether or not a synchronization signal has been received. If the result of the determination is that a synchronization signal has been received, the process proceeds to S1003 and S1007. On the other hand, if a synchronization signal has not been received, the process returns to S1001 and performs the same processing cycle.

Next, in S1003, an instruction is issued to the imaging sensor of the imaging unit 206 to capture an image. Next, in S1004, the imaging unit 206 performs image processing at the timing output from the synchronization signal transmission device 106, and generates a video 310 captured at the same timing within the width of the frame time. Next, in S1005, the video signal output unit 207 converts the video 310 image-processed by the imaging unit 206 into a video signal, and then outputs it to the information processing device 102. After that, the process returns to S1001 and the same process is periodically performed.

Next, in S1007, the information holding unit 204 acquires pan-tilt information from the pan-tilt unit 203, and acquires the pan-tilt information acquisition time from the time providing unit 205 on the camera side. Next, in S1008, the information holding unit 204 transmits the pan-tilt information and the pan-tilt information acquisition time to the packet generating unit 208 by in-flight communication. Thereafter, the process returns to S1001 and the same process is performed periodically. Note that it is preferable that the timing of starting the processes of S1003 and S1007 is linked, but it is more preferable that the processes of S1003 and S1007 are started simultaneously.

The processing flow of the communication processing unit 202 shown in FIG. 6 shows the process from receiving pan/tilt information from the imaging processing unit 201 via in-flight communication to outputting it as a packet. And, like the imaging processing unit 201 shown in FIG. 6, the entire process is performed periodically.

First, in S1009, the time providing unit 209 on the network side synchronizes time with the imaging processing unit 201 every cycle (predetermined cycle). Next, in S1010, it is determined whether or not the packet generating unit 208 has received the pan-tilt information and the pan-tilt information acquisition time by in-flight communication. If the result of the determination is that the packet generating unit 208 has received the pan-tilt information and the pan-tilt information acquisition time by in-flight communication, the process proceeds to S1011. On the other hand, if the packet generating unit 208 has not received the pan-tilt information and the pan-tilt information acquisition time by in-flight communication, the process returns to S1009 and the same process is performed periodically.

Next, in S1011, the packet generating unit 208 converts the received pan-tilt information into a packet format that can be received by the information processing device 102 via the network 103, and generates a packet. At this time, the pan-tilt information acquisition time is not included in the packet that contains the pan-tilt information, and the packet is transmitted to the time adding unit 210.

Next, in S1012, the time addition unit 210 acquires the packet sending time from the time provision unit 209 on the network side. Next, in S1013, the time addition unit 210 calculates the differential time 401 based on the packet sending time acquired in S1012 and the pan/tilt information acquisition time transmitted from the packet generation unit 208 in S1011.

Next, in S1014, it is determined whether the difference time 401 calculated in S1013 is greater than the time between one frame of the video currently being output. If the result of the determination is that the difference time 401 is greater than the time between one frame of the video currently being output, the process proceeds to S1015. On the other hand, if the difference time 401 is less than the time between one frame of the video currently being output, the process proceeds to S1016.

Next, in S1015, the time addition unit 210 adds the differential time 401 to the packet to which the pan/tilt information has been added. Next, in S1016, the time addition unit 210 transmits the packet to which the differential time has not been added to the information processing device 102. Thereafter, the process returns to S1009 and the same process is periodically performed.

Here, the processing flow of the information processing device 102 shown in FIG. 7 shows the process up to generating a synchronized composite image using the image 310 transmitted from the imaging processing unit 201 and the pan-tilt information to which the difference time 401 transmitted from the communication processing unit 202 is added.

First, in S1017, the information processing device 102 receives the image 310 from the image signal output unit 207 of the imaging processing unit 201. Next, in S1018, the image 310 received in S1017 is stored in the image storage unit 215. Next, in S1019, the communication unit 212 receives the pan-tilt information transmitted from the time addition unit 210. Thereafter, the communication unit 212 transmits the received pan-tilt information to the pan-tilt information drawing unit 213.

Next, in S1020, the three-dimensional graphic generation unit 214 generates a three-dimensional graphic and transmits the generated three-dimensional graphic data to the pan/tilt information drawing unit 213. Next, in S1021, the pan/tilt information drawing unit 213 performs rendering on the three-dimensional graphic data while reflecting the pan/tilt information transmitted from the communication unit 212, and generates a three-dimensional graphic image.

Next, in S1022, it is determined whether or not the differential time 401 has been added to the pan-tilt information. If the differential time 401 has been added to the pan-tilt information, the process proceeds to S1023. On the other hand, if the differential time 401 has not been added to the pan-tilt information, the process proceeds to S1025.

Next, in S1023, the drawing synthesis unit 216 acquires the difference time 401 corresponding to the pan/tilt information used for drawing from the pan/tilt information drawing unit 213. Next, in S1024, the drawing synthesis unit 216 selects the image from the image storage unit 215 that is older than the difference time 401, i.e., the image 310 that dates back in time. Next, in S1025, since the difference time 401 has not been added to the pan/tilt information, the latest image 310 is selected.

Next, in S1026, the drawing synthesis unit 216 synthesizes the rendered three-dimensional graphic image and the selected image 310 to generate a synthetic image. Next, in S1027, since the past image 310 stored in the image storage unit 215 becomes unnecessary at the stage of generating the synthetic image in S1026, the selected older image 310 is deleted. Next, in S1028, the synthetic image generated in S1026 is output as a result to an external device or the like. Note that the output synthetic image may be displayed in any format, for example, on the display (display unit) 104. This allows the user to view a synthetic image that is free of temporal difference and has the same pan-tilt situation as the image when synthesized.

Since the images of the first imaging device 100 and the second imaging device 105, whose difference times are the same, are synchronized by a signal from the synchronization signal transmitter 106, it is possible to synchronize the respective images and pan/tilt information by performing similar processing in the information processing device 102.

By performing the above process, it is possible to synchronize the images of each imaging device, which is synchronized by the signal from the synchronization signal transmitter 106, with the three-dimensional graphic image generated based on the pan-tilt information output by the packet. Then, by synthesizing the images with the three-dimensional graphic image, it is possible to provide a system that can generate a composite image with the same pan-tilt situation as the image, without any time difference.

<Embodiment 2>
Next, a system according to a second embodiment of the present invention will be described with reference to Fig. 8 to Fig. 19. In the second embodiment, pan-tilt information is shared between each imaging device, such as the first imaging device 100 and the second imaging device 105, so that the information processing device 102 synchronizes the image 310 with the pan-tilt information without considering the difference time 401 in each imaging device.

In the second embodiment, the same components as those in the first embodiment are designated by the same reference numerals already used, and detailed description thereof will be omitted. In the second embodiment, the description will focus on the captured image storage unit 801, the packet receiving unit 802, the delayed image signal output unit 803, and the second time addition unit (second pan/tilt time addition unit) 804, which performs control different from that in the first embodiment.

Figures 8 and 9 are diagrams showing the system configuration in the second embodiment. Specifically, Figure 8 is a diagram showing a system configuration in which there is a difference (difference) in the differential time between imaging devices, and Figure 9 is a diagram showing the system configuration of an imaging device and an information processing device in which the differential time is synchronized. In Figure 8, a third imaging device 601, which differs from the configuration in Figure 1 in the first embodiment, is explained by showing the differences with the first imaging device 100 and the second imaging device 105.

In FIG. 8, the differential time in the first imaging device 100 is shown as the first differential time 602, the differential time in the second imaging device 105 is shown as the second differential time 603, and the differential time in the third imaging device 601 is shown as the third differential time 604. Note that in the following, in the systems shown in FIG. 8 and FIG. 9, the first differential time 602, the second differential time 603, and the third differential time 604 of each imaging device will be described as being significantly different.

When there are a large number of imaging devices connected to the information processing device 102 and their differential times 401 are different, the number of objects to be synchronized in the information processing device 102 increases. In this case, the information processing device 102 receives the differential time 401 of the pan/tilt information, and by delaying the images 310 by a certain amount and synthesizing them, it is possible to synchronize the images 310 and pan/tilt information of each imaging device. However, the more imaging devices are connected to the information processing device 102, the more difficult it becomes to synchronize all of the images 310 and pan/tilt information between the imaging devices.

Therefore, in the system of this embodiment, all imaging devices delay the packet output and image output to match the imaging device with the largest differential time among the multiple imaging devices. At this time, each imaging device does not add the calculated differential time 401 to the packet as is, but transmits its own differential time to each imaging device as a new actual differential time 704, and is controlled so that the actual differential time 704 is shared between each imaging device. After that, each imaging device determines the largest actual differential time 704 among the received actual differential times 704 as the differential time 401 to be added to the packet that also contains pan/tilt information. Then, the determined actual differential time 704 is added to the packet and transmitted to the information processing device 102.

At this time, when transmitting the packet to which the actual difference time 704 is added to the information processing device 102, the difference between the actual difference time 704 and the difference time 401 calculated by the imaging device is calculated, and the packet and the output of the image 310 are delayed by the difference and transmitted to the information processing device 102. The image 310 is then transmitted to the information processing device 102 with a delay of the largest difference time (maximum difference time). This makes it possible for the information processing device 102 to obtain pan/tilt information and the image 310 in a synchronized state.

Figure 9 shows a case where the third differential time 604 in the third imaging device 601 is the largest differential time. Then, the first differential time 602 in the first imaging device 100 is the next largest differential time, and the second differential time 603 in the second imaging device 105 is the smallest differential time.

In this case, the first imaging device 100 transmits the first differential time 602 as the actual differential time 704 to the second imaging device 105 and the third imaging device 601. The second imaging device 105 transmits the second differential time 603 as the actual differential time 704 to the first imaging device 100 and the third imaging device 601. The third imaging device 601 transmits the third differential time 604 as the actual differential time 704 to the first imaging device 100 and the second imaging device 105. This allows the actual differential times of each imaging device to be shared between each imaging device.

Then, each imaging device adds the actual difference time 704 (third difference time 604) of the third imaging device 601, which is the largest difference time, to the packet to be transmitted to the information processing device 102. That is, the first imaging device 100 adds the actual difference time 704 to the packet 701 in the first imaging device. The second imaging device 105 adds the actual difference time 704 to the packet 702 in the second imaging device. The third imaging device 601 adds the actual difference time 704 to the packet 703 in the third imaging device. Then, when transmitting each packet to the information processing device 102, each imaging device calculates the difference between the actual difference time 704 and the difference time calculated by that imaging device, and delays the output of the packet and the video 310 by the difference and transmits them to the information processing device 102. Also, the video 310 in each imaging device is transmitted to the information processing device 102 with a delay of the actual difference time 704. This allows each imaging device to delay the output of the video 310 and packets.

Fig. 10 is a diagram showing functional blocks of a system according to the second embodiment. Specifically, an example of functional blocks of the first imaging device 100 and the second imaging device 105 is shown. Note that, as in the first embodiment, the functions of each imaging device and the information processing device 102 shown in Fig. 8 are realized by the CPU of the first imaging device 100 or the second imaging device 105 executing a computer program stored in a ROM.

The packet receiving unit 802 receives the actual difference time 704 of the other imaging device from the communication unit 211 on the imaging device side, and receives the difference time 401, which is its own actual difference time, from the second time adding unit 804. Then, the packet receiving unit 802 compares the actual difference time 704 acquired from the other imaging device with the difference time 401. As a result of the comparison, the packet receiving unit 802 transmits the largest difference time to the second time adding unit 804 and the captured image holding unit 801.

The second time adding unit 804 adds the largest differential time transmitted from the packet receiving unit 802 to the packet as the differential time to be added to the packet, and transmits the packet. When transmitting the packet, the second time adding unit 804 calculates the difference between the largest differential time and its own differential time 401, and outputs the packet with a delay of the calculated difference. In order to delay the packet, the second time adding unit 804 holds pan-tilt information for multiple times, selects pan-tilt information of an appropriate time, and transmits the selected information as a packet.

The control of the imaging unit 206 is the same as in the first embodiment, but the image 310 is transmitted to the captured image holding unit 801 instead of the image signal output unit 207. The captured image holding unit 801 receives the largest difference time from the packet receiving unit 802, and transmits the image 310 delayed by the largest difference time to the delayed image signal output unit 803. The delayed image signal output unit 803 converts the image delayed by the largest difference time transmitted from the captured image holding unit 801 into a video signal and outputs it to the information processing device 102.

For example, if two or more second imaging devices 105 and third imaging devices 601 also have the same configuration, it becomes possible to transmit the image 310 and pan/tilt information synchronized with the largest difference time to the information processing device 102. Note that while FIG. 8 shows the configurations of the first imaging device 100 and second imaging device 105 as an example, similar synchronization can also be achieved with the third imaging device 601 and other different imaging devices having the same configuration.

Figures 11, 12, and 13 show a case where each imaging device shown in Figure 8 operates with a different differential time 401 and performs the synchronization process described in embodiment 1. Figure 14 shows the time correlation of control in an information processing device when there is a difference in differential time. Figures 11, 12, and 13 show a state where the third differential time 604 of the third imaging device 601 is longer than the first differential time 602 of the first imaging device 100 by one frame or more.

Furthermore, the second differential time 603 of the second imaging device 105 is shorter than the first differential time 602 of the first imaging device 100 by one frame or more. In this case, it is assumed that each differential time of each imaging device is held and synchronization processing is performed by the information processing device 102. At this time, the frame numbers of the composite image 901 of the first imaging device 100, the composite image 902 of the second imaging device 105, and the composite image 903 of the third imaging device 601 generated by the information processing device 102 are each shifted by one frame or more. This results in a state where the composite images are not synchronized.

Figures 15, 16, and 17 are diagrams showing the time relationships when each imaging device delays and outputs packets and video corresponding to the largest time difference in each imaging device. Time t in Figures 15, 16, and 17 indicates the delayed video processing time 1001, which indicates the time when the video stored in the captured video storage unit 801 is read and output by the delayed video signal output unit 803. Note that because synchronization processing is performed within each imaging device 100, a description of the information processing device 102 is omitted.

In Figures 15, 16, and 17, in addition to the video signal path 308, delayed video 1002 is shown. Furthermore, instead of the pan/tilt information path 309, delayed pan/tilt information path 1003 is shown. The delayed video 1002 shows how it is output from the delayed video signal output unit 803 after being delayed by the time of the video signal path 308 plus the third differential time 604 of the third imaging device 601. By outputting the video 310 according to the largest differential time, the frames of the video 310 match between the first to third imaging devices.

The delayed pan/tilt information path 1003 shows how the packet output is delayed to match the largest time difference. By outputting the image 310 to match the largest time difference, the packet output timing is synchronized in the first to third imaging devices. By transmitting the image 310 and the pan/tilt information in the packet with a timing that matches the largest time difference in each imaging device through the above processing, it becomes possible to transmit the pan/tilt information and the image in synchronization with each other to the information processing device 102.

In the process shown in FIG. 18 and FIG. 19, the actual difference time 704 of each imaging device is shared between the imaging devices, and the actual difference time with the largest difference time is determined as the difference time 410 to be added to the packet as the largest difference time. Then, within each imaging device, the video 310 and the pan/tilt information packet are output with a delay corresponding to the largest difference time. As a result, both the video 310 and the pan/tilt information are transmitted to the information processing device 102 with a delay corresponding to the largest difference time, so the information processing device 102 can synchronize the video 310 and the pan/tilt information without internally storing the video 310.

Fig. 18 is a diagram showing the processing flow of the imaging processing unit 201 according to the second embodiment. Fig. 19 is a diagram showing the processing flow of the communication processing unit 202 according to the second embodiment. Note that the information processing device 102 receives the synchronized image 310 and pan/tilt information as described above and does not require internal processing, so the processing flow of the information processing device 102 will be omitted. Also, by adding an S to the beginning of each process (step), the notation of the process (step) will be omitted.

The processing flow of the imaging processing unit 201 shown in FIG. 18 describes the process up to outputting the image 310 converted into a video signal and outputting the pan/tilt information to the packet generating unit 208. The entire process shown in FIG. 18 is performed periodically.

First, in S1001, the time providing unit 205 on the camera side performs time synchronization with the communication processing unit 202 every cycle (predetermined cycle). Next, in S1002, it is determined whether or not a synchronization signal has been received. If the result of the determination is that a synchronization signal has been received, the process proceeds to S1003 and S1007. On the other hand, if a synchronization signal has not been received, the process returns to S1001 and performs the same processing cycle.

Next, in S1003, an instruction to capture an image is issued to the imaging sensor of the imaging device. Next, in S1004, the imaging unit 206 performs image processing at the timing output from the synchronization signal transmission device 106, generates an image 310 captured at the same timing within the width of the frame time, and outputs the image to the captured image storage unit 801.

Next, in S1101, the captured image storage unit 801 stores the image 310 output from the imaging unit 206. Next, in S1102, the captured image storage unit 801 acquires the largest differential time from the packet receiving unit 802. Next, in S1103, from among the multiple images 310 stored in the captured image storage unit 801, an image that is older than the largest differential time, i.e., an image that goes back in time, is selected.

Next, in S1104, the image 310 selected in S1102 is output from the delayed image signal output unit 803 to the information processing device 102. Next, in S1105, images recorded before the image 310 selected in S1102 are deleted from the captured image storage unit 801. After that, the process returns to S1001 and the same processing is performed periodically.

Next, in S1007, the information holding unit 204 acquires pan/tilt information related to pan/tilt from the pan/tilt unit 203, and acquires the pan/tilt information acquisition time from the time providing unit 205 on the camera side. Next, in S1008, the information holding unit 204 transmits the pan/tilt information and the pan/tilt information acquisition time to the packet generating unit 208 by in-flight communication. Thereafter, the process returns to S1001 and the same process is performed periodically. Note that it is preferable that the timing of starting the processes of S1003 and S1007 is linked, but it is more preferable that the processes of S1003 and S1007 are started simultaneously.

The processing flow of the communication processing unit 202 shown in FIG. 19 shows the process of receiving packets from other imaging devices and finding the largest difference time, and the process of receiving pan/tilt information from the imaging processing unit 201 via in-flight communication and outputting it as a packet. The entire process shown in FIG. 19 is performed periodically.

First, in S1009, the time providing unit 209 on the network side synchronizes time with the imaging processing unit 201 every cycle (predetermined cycle). Next, in S1111, it is determined whether or not a differential time has been acquired from another imaging device. If a differential time has been acquired from another imaging device, the process proceeds to S1112. On the other hand, if a differential time has not been acquired from another imaging device, the process returns to S1009 and performs the same process. The differential time from another imaging device here refers to the actual differential time 704 in the other imaging device.

Next, in S1112, a comparison is made to see whether the difference time acquired from the other imaging device is greater than the device's own difference time. If the comparison result shows that the difference time acquired from the other imaging device is greater than the device's own difference time, the process proceeds to S1113. On the other hand, if the difference time acquired from the other imaging device is smaller than the device's own difference time, the process returns to S1009 and the same process is periodically performed.

Next, in S1112, the differential time is updated. This allows the system to determine which of the other devices has the largest differential time, and each imaging device can store the largest differential time between the connected imaging devices.

Next, in S1010, it is determined whether or not the packet generation unit 208 has received the pan-tilt information and the pan-tilt information acquisition time via in-flight communication. If the result of the determination is that the packet generation unit 208 has received the pan-tilt information and the pan-tilt information acquisition time via in-flight communication, the process proceeds to S1011. On the other hand, if the packet generation unit 208 has not received the pan-tilt information and the pan-tilt information acquisition time via in-flight communication, the process returns to S1009 and the same process is periodically performed.

Next, in S1006, the pan/tilt information received via in-flight communication is stored. Next, in S1012, the time addition unit 210 acquires the packet sending time from the time provision unit 209 on the network side. Next, in S1013, the time addition unit 210 calculates the differential time 401 based on the packet sending time acquired in S1012 and the pan/tilt information acquisition time sent from the packet generation unit 208 in S1011.

Next, in S1107, a comparison is made to see whether the difference time 401 calculated in S1013 is greater than the largest difference time. If the comparison result shows that the difference time 401 calculated in S1013 is greater than the largest difference time, the process proceeds to S1108. On the other hand, if the difference time 401 calculated in S1013 is less than the largest difference time, the process proceeds to S1109.

Next, in S1108, the difference time is updated so that the difference time 401 calculated in S1013 becomes the largest difference time. Next, in S1109, the difference between the largest difference time and the own difference time 401 is calculated, and pan/tilt information from before the calculated time is obtained.

Next, in S1011, the packet generating unit 208 converts the received pan-tilt information into a packet format that can be received by the information processing device 102 via the network 103, and generates a packet. At this time, the pan-tilt information acquisition time is not included in the packet that contains the pan-tilt information, and the packet is transmitted to the time adding unit 210.

Next, in S1110, the second time addition unit 804 adds the largest differential time to the packet to which the pan/tilt information has been added. Next, in S1016, the second time addition unit 804 transmits the packet to which the largest differential time has been added to the information processing device 102. Next, in S1114, the pan/tilt information held before the pan/tilt information acquired in S1109 is deleted. After that, the process returns to S1009 and the same process is periodically performed.

According to the above processing, each imaging device can output the image 310 of the imaging device synchronized by the signal from the synchronization signal transmission device 106 and the pan/tilt information output by the packet based on the same difference time. Therefore, the information processing device 102 can receive the image 310 and the pan/tilt information in a time-synchronized state. By synthesizing the 3D graphic image generated in the above state with the image 310 of the imaging device, it is possible to provide a system that can output the images of multiple imaging devices and the 3D graphic image in a synchronized manner.

In the first and second embodiments, the delayed video signal output unit 803 and the video signal output unit 207 are described as being different, but they may be configured as a single output unit having both of the above-mentioned functions. Furthermore, each imaging device in the first and second embodiments may be equipped with either the delayed video signal output unit 803 or the video signal output unit 207. In that case, the configuration may be such that it is selectable whether or not to delay the video 310 or packets. Furthermore, a plurality of these configurations may be held, and the output method of the video 310 to be output may be selectable or switchable by user operation.

In the above embodiments, the focus was on panning and tilting, but the angle of view can also be synchronized by adding information about the motor drive system, which affects the angle of view of the image, to the pan-tilt information in a similar manner.

The disclosure of this embodiment includes the following configuration, method, and program.

(Configuration 1)
an imaging unit that performs image processing on an image captured in synchronization with a signal output from a synchronization signal transmission device that transmits a periodic signal, thereby generating a video;
A pan-tilt unit capable of changing the angle of view of the imaging unit;
an output unit that outputs the image generated by the imaging unit;
a packet generating unit that acquires status information that is information on the status of the pan/tilt unit when the signal output from the synchronization signal transmitting device is detected, and generates a packet including the status information;
a time adding unit that calculates a difference time between a first time that is a time when the state information is acquired and a second time that is a time when the packet generating unit outputs the packet, and adds the difference time to the packet.
1. An imaging device comprising:

(Configuration 2)
A time providing unit that provides a synchronized time;
2. The imaging device according to configuration 1, wherein the time adding section obtains the second time from a time providing section.

(Configuration 3)
3. The imaging device according to configuration 2, further comprising an information storage unit for storing the state information and the first time.

(Configuration 4)
4. The imaging device according to configuration 3, wherein the information holding unit obtains the first time from the time providing unit and the state information from the time providing unit when the output unit outputs the video.

(Configuration 5)
5. The imaging device according to configuration 3 or 4, wherein the information holding section detects the signal output from the synchronization signal transmission device by notification from the imaging section, and acquires the state information.

(Configuration 6)
6. The imaging device according to any one of configurations 1 to 5, wherein the output section outputs an image in accordance with a signal output from a synchronization signal transmission device.

(Configuration 7)
The imaging device described in any one of configurations 1 to 6, characterized in that the time addition unit adds the differential time to the packet when the differential time is a time greater than one frame in the video, and does not add the differential time to the packet when the differential time is a time smaller than one frame in the video.

(Configuration 8)
8. The imaging device according to any one of configurations 1 to 7, further comprising a communication unit that transmits a packet to which a difference time has been added to another imaging device different from the imaging device itself.

(Configuration 9)
9. The imaging device according to configuration 8, wherein the other imaging device is at least one.

(Configuration 10)
10. The imaging device according to configuration 8 or 9, further comprising a packet receiving unit for receiving a packet to which a difference time in another imaging device is added.

(Configuration 11)
11. The imaging device according to configuration 10, wherein the packet receiving section determines the largest differential time by comparing the differential time in the imaging device with the differential time in the other imaging devices.

(Configuration 12)
12. The imaging device according to configuration 11, wherein the time adding section adds the largest difference time to the packet.

(Configuration 13)
13. The imaging device according to configuration 11 or 12, wherein the time adding section calculates the difference between the largest differential time and the differential time, and outputs a packet according to the calculated difference.

(Configuration 14)
An image storage unit that stores an image,
14. The imaging device according to claim 13, wherein the image storage unit outputs the image delayed by the largest difference time.

(Configuration 15)
15. The imaging device according to configuration 14, wherein the output unit outputs the video output by the video holding unit, delayed by the largest time difference, in synchronization with the signal output by the synchronization signal transmission device.

(Configuration 16)
16. The imaging device according to any one of configurations 1 to 15, wherein the state information is information about a current angle of view of the imaging unit.

(Configuration 17)
An image is captured in accordance with a signal output from a synchronization signal transmitter that transmits a periodic signal, and image processing is performed on the captured image to generate a video image.
Output the generated image,
acquiring status information that is information on a status of a pan-tilt unit capable of changing the angle of view of an imaging unit that generates an image when a signal output from a synchronization signal transmission device is detected, and generating a packet including the status information;
calculating a differential time between a first time which is a time when the status information is acquired and a second time which is a time when the packet is output, and adding the differential time to the packet;
23. A method for controlling an imaging apparatus comprising:

(Configuration 18)
A program for causing a computer to execute a control method for an imaging device,
An image is captured in accordance with a signal output from a synchronization signal transmitter that transmits a periodic signal, and image processing is performed on the captured image to generate a video image.
Output the generated image,
acquiring status information that is information on a status of a pan-tilt unit capable of changing the angle of view of an imaging unit that generates an image when a signal output from a synchronization signal transmission device is detected, and generating a packet including the status information;
calculating a differential time between a first time which is a time when the status information is acquired and a second time which is a time when the packet is output, and adding the differential time to the packet;
A program characterized by:

(Configuration 19)
a communication unit that receives a packet to which a difference time is added, the difference being between a first time, which is a time at which state information is acquired as information on a state of a pan-tilt unit of the imaging device when a signal output from a synchronization signal transmission device that transmits a periodic signal is detected, and a second time, which is a time at which the packet is output;
a storage unit that stores a video image generated by performing image processing on the image captured in accordance with the signal;
a generating unit for generating a graphic corresponding to the video;
a drawing unit that changes the drawing of a graphic based on the state information;
a drawing synthesis unit that synthesizes the graphic drawn by the drawing unit with the video to generate a synthesized video;
the drawing synthesis unit, when the difference time added to the packet is equal to or greater than a predetermined time, selects an image that is older than the difference time from the images stored in the storage unit based on the difference time, and synthesizes the selected image with the graphic drawn by the drawing unit to generate a synthetic image;
23. An information processing apparatus comprising:

(Configuration 20)
An imaging system having at least one imaging device and an information processing device communicably connected to the imaging device,
The imaging device
an imaging unit that performs image processing on an image captured in synchronization with a signal output from a synchronization signal transmission device that transmits a periodic signal, thereby generating a video;
A pan-tilt unit capable of changing the angle of view of the imaging unit;
an output unit that outputs the image generated by the imaging unit;
a packet generating unit that acquires status information that is information on the status of the pan/tilt unit when the signal output from the synchronization signal transmitting device is detected, and generates a packet including the status information;
a time adding unit that calculates a difference time between a first time that is a time when the state information is acquired and a second time that is a time when the packet generating unit outputs a packet, and adds the difference time to the packet,
The information processing device includes:
A communication unit for receiving packets;
A storage unit for storing an image;
A generating unit that generates a predetermined graphic corresponding to the video;
a drawing unit that changes the drawing of a predetermined graphic based on the state information;
a drawing synthesis unit that synthesizes the graphic drawn by the drawing unit with the video to generate a synthesized video;
The imaging system is characterized in that, when the differential time added to the packet is equal to or greater than a predetermined time, the drawing synthesis unit selects an image from the image stored in the storage unit that was taken before the differential time based on the differential time, and synthesizes the selected image with the graphic drawn by the drawing unit to generate a composite image.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. In this case, the program and the storage medium on which the program is stored constitute the present invention. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

REFERENCE SIGNS LIST 100 First imaging device 102 Information processing device 103 Network 104 Display 105 Second imaging device 106 Synchronization signal transmission device 201 Imaging processing unit 202 Communication processing unit 203 Pan/tilt unit 204 Storage unit 205 Time providing unit 206 Imaging unit 207 Video signal output unit 208 Packet generation unit 209 Time providing unit 210 Time addition unit 211 Imaging device communication unit 212 Display device communication unit 213 Pan/tilt information drawing unit 214 Three-dimensional graphics generation unit 215 Video storage unit 216 Drawing synthesis unit

Claims (20)

an imaging unit that performs image processing on an image captured in synchronization with a signal output from a synchronization signal transmission device that transmits a periodic signal, thereby generating a video;
A pan-tilt unit capable of changing the angle of view of the imaging unit;
an output unit that outputs the image generated by the imaging unit;
a packet generating unit that acquires status information that is information on the status of the pan/tilt unit when the signal output from the synchronization signal transmitting device is detected, and generates a packet including the status information;
a time adding unit that calculates a difference time between a first time that is a time when the state information is acquired and a second time that is a time when the packet generating unit outputs the packet, and adds the difference time to the packet.
1. An imaging device comprising: A time providing unit that provides a synchronized time;
The imaging device according to claim 1 , wherein the time adding unit obtains the second time from the time providing unit. The imaging device according to claim 2, further comprising an information storage unit that stores the state information and the first time. The imaging device according to claim 3, characterized in that the information storage unit acquires the first time from the time providing unit and the status information from the time providing unit when the output unit outputs the video. The imaging device according to claim 3, characterized in that the information storage unit detects the signal output by the synchronization signal transmission device by notification from the imaging unit and acquires the status information. The imaging device according to claim 1, characterized in that the output unit outputs the video in accordance with the signal output by the synchronization signal transmission device. The imaging device according to claim 1, characterized in that the time addition unit adds the differential time to the packet when the differential time is a time longer than one frame in the video, and does not add the differential time to the packet when the differential time is a time shorter than one frame in the video. The imaging device according to claim 1, further comprising a communication unit that transmits the packet to which the difference time has been added to an imaging device other than the imaging device. The imaging device according to claim 8, characterized in that the other imaging device is at least one. The imaging device according to claim 8, further comprising a packet receiving unit for receiving packets to which the difference time in the other imaging device is added. The imaging device according to claim 10, characterized in that the packet receiving unit determines the largest difference time by comparing the difference time in the other imaging device with the difference time in the imaging device. The imaging device according to claim 11, characterized in that the time addition unit adds the largest difference time to the packet. The imaging device according to claim 11, characterized in that the time addition unit calculates the difference between the largest differential time and the differential time, and outputs the packet according to the calculated difference. an image storage unit for storing the image;
14. The imaging device according to claim 13, wherein the image storage unit outputs the image delayed by the largest time difference. The imaging device according to claim 14, characterized in that the output unit outputs the image delayed by the largest difference time output by the image storage unit in accordance with the signal output by the synchronization signal transmission device. The imaging device according to claim 1, characterized in that the status information is information about the current angle of view of the imaging unit. An image is captured in accordance with a signal output from a synchronization signal transmitter that transmits a periodic signal, and image processing is performed on the captured image to generate a video image.
Outputting the generated image;
acquiring status information that is information on a status of a pan-tilt unit capable of changing the angle of view of an imaging unit that generates the image when the signal output by the synchronization signal transmission device is detected, and generating a packet including the status information;
calculating a differential time between a first time which is a time when the state information is acquired and a second time which is a time when the packet is output, and adding the differential time to the packet;
23. A method for controlling an imaging apparatus comprising the steps of: A program for causing a computer to execute a control method for an imaging device,
An image is captured in accordance with a signal output from a synchronization signal transmitter that transmits a periodic signal, and image processing is performed on the captured image to generate a video image.
Outputting the generated image;
acquiring status information that is information on a status of a pan-tilt unit capable of changing the angle of view of an imaging unit that generates the image when the signal output by the synchronization signal transmission device is detected, and generating a packet including the status information;
calculating a differential time between a first time which is a time when the state information is acquired and a second time which is a time when the packet is output, and adding the differential time to the packet;
A program characterized by: a communication unit that receives a packet to which a difference time is added, the difference being between a first time, which is a time at which state information is acquired as information on a state of a pan-tilt unit of the imaging device when a signal output from a synchronization signal transmission device that transmits a periodic signal is detected, and a second time, which is a time at which the packet is output;
a storage unit that stores a video image generated by performing image processing on the image captured in accordance with the signal;
a generating unit for generating a graphic corresponding to the image;
a drawing unit that changes the drawing of the graphic based on the state information;
a drawing synthesis unit that synthesizes the graphic drawn by the drawing unit and the image to generate a synthetic image,
when the differential time added to the packet is equal to or greater than a predetermined time, the drawing synthesis unit selects an image that is older than the differential time from the images stored in the storage unit based on the differential time, and synthesizes the selected image with the graphic drawn by the drawing unit to generate a synthetic image.
23. An information processing apparatus comprising: An imaging system having at least one imaging device and an information processing device communicably connected to the imaging device,
The imaging device includes:
an imaging unit that performs image processing on an image captured in synchronization with a signal output from a synchronization signal transmission device that transmits a periodic signal, thereby generating a video;
A pan-tilt unit capable of changing the angle of view of the imaging unit;
an output unit that outputs the image generated by the imaging unit;
a packet generating unit that acquires status information that is information on the status of the pan/tilt unit when the signal output from the synchronization signal transmitting device is detected, and generates a packet including the status information;
a time adding unit that calculates a difference time between a first time that is a time when the state information is acquired and a second time that is a time when the packet generating unit outputs the packet, and adds the difference time to the packet,
The information processing device includes:
A communication unit for receiving the packet;
A storage unit for storing the image;
a generating unit for generating a predetermined graphic corresponding to the image;
a drawing unit that changes the drawing of the predetermined graphic based on the state information;
a drawing synthesis unit that synthesizes the graphic drawn by the drawing unit and the image to generate a synthetic image,
The imaging system is characterized in that, when the differential time added to the packet is equal to or greater than a predetermined time, the drawing synthesis unit selects, based on the differential time, an image from the image stored in the storage unit that was taken before the differential time, and synthesizes the selected image with the graphic drawn by the drawing unit to generate a synthetic image.

Images