JP2008193561A - Image synchronization system and image synchronizing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily and synchronously reproduce a plurality of images. <P>SOLUTION: The image synchronization system is equipped with two imaging devices 1 which synchronize and reproduce images associated with one scene, where the imaging device includes a memory 5 to record two digital audio data recorded during imaging by the two imaging devices, an audio synchronism detector 8 to acquire time difference information for synchronously reproducing two images based upon the two digital audio data, and a CPU 121 to control the reproduction timing of the image by the first imaging device based upon the time difference information, and the second imaging device includes a communication unit 7 to receive the time difference information transmitted from the first imaging device and a CPU 121 to control the reproduction timing of the image by the second imaging device. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image synchronization system and an image synchronization method for synchronizing images reproduced by a plurality of reproduction apparatuses.

2. Description of the Related Art Conventionally, there has been developed an apparatus that captures and records a scene such as sports with a plurality of imaging devices installed with different viewpoints, and reproduces the recorded moving images as two screens on a display in synchronization. Specifically, a plurality of moving images are accumulated on a hard disk as a plurality of channel data files, and a synchronization start position is set based on data recorded in an audio track portion of the plurality of channel data files. A plurality of moving images are synchronized and reproduced according to the start position (see, for example, Patent Document 1).
JP 2001-36867 A

However, in the case of Patent Document 1 or the like, since a dedicated multi-channel moving image synchronous display device is required, there is a problem that versatility is low and synchronized reproduction of a plurality of moving images cannot be easily performed.
In addition, in order to set the synchronization start position in a plurality of moving images, it is necessary to generate and record a start signal with a whistle, an electronic sound generator, or the like after the start of shooting, which causes a problem that the operation is troublesome.

  Therefore, an object of the present invention is to provide an image synchronization system and an image synchronization method that can easily perform synchronized reproduction of a plurality of images.

The invention described in claim 1
An image synchronization system comprising a plurality of playback devices that play back images related to one scene in synchronization,
Each of the plurality of playback devices is
Image recording means for recording any one of a plurality of images obtained by capturing the one scene with a plurality of imaging means;
Image reproducing means for reproducing an image recorded in the image recording means,
Among the plurality of playback devices, at least one playback device is:
A plurality of sound recording means for recording a plurality of sound information recorded in association with a plurality of images picked up by the plurality of image pickup means;
Image synchronization information acquisition for acquiring image synchronization information for synchronizing and reproducing a plurality of images by the image reproduction means of the plurality of reproduction devices based on the plurality of audio information recorded in the plurality of audio recording means Means,
First reproduction control means for controlling the reproduction timing of the image by the image reproduction means based on the image synchronization information acquired by the image synchronization information acquisition means;
Synchronization information transmitting means for transmitting the image synchronization information acquired by the image synchronization information acquiring means to another playback device different from the one playback device among the plurality of playback devices,
The other playback device is:
Synchronization information receiving means for receiving the image synchronization information transmitted from the synchronization information transmitting means;
And a second reproduction control unit that controls the reproduction timing of the image by the image reproduction unit based on the image synchronization information received by the synchronization information reception unit.

The invention according to claim 2 is the image synchronization system according to claim 1,
The image synchronization information acquisition means includes
Deviation information search means for searching for a plurality of deviation information taking a maximum value of a cross-correlation function within a predetermined search range of the plurality of voice information;
Image synchronization information calculating means for calculating an average value of the plurality of deviation information searched by the deviation information searching means and making the image synchronization information,
The image synchronization information calculation means calculates the average value using the maximum value of the cross-correlation function as a weighting coefficient.

The invention according to claim 3 is the image synchronization system according to claim 1 or 2,
The one playback device is:
Based on the image synchronization information acquired by the image synchronization information acquisition means, an image group whose imaging has been started later is selected by the image reproduction means from among the plurality of image groups captured by the plurality of imaging means. Priority information acquisition means for acquiring priority image group identification information as a priority image group for starting playback from:
An image group whose reproduction ends early after the reproduction of the priority image group by the image reproduction unit among the plurality of image groups reproduced in synchronization by the image reproduction unit based on the plurality of audio information An end information acquisition means for acquiring early end image group identification information,
Specific information transmitting means for transmitting the priority image group specifying information acquired by the priority information acquiring means and the early end image group specifying information acquired by the end information acquiring means to the other playback device,
The first reproduction control unit is further configured to reproduce the image based on the priority image group specifying information acquired by the priority information acquiring unit and the early end image group specifying information acquired by the end information acquiring unit. Controlling the reproduction timing of the image group by the means so as to reproduce the duplicate images captured by the plurality of imaging means in a synchronized manner,
The other playback device is:
Specific information receiving means for receiving the priority image group specifying information and the early ending image group specifying information transmitted from the specific information transmitting means,
The second reproduction control unit further sets the reproduction timing of the image by the image reproduction unit based on the priority image group identification information and the early end image group identification information received by the identification information reception unit. It is characterized by controlling to play back images synchronously.

The invention according to claim 4 is the image synchronization system according to any one of claims 1 to 3,
The other playback device is:
Other audio recording means for recording audio information recorded in association with the one image recorded in the image recording means;
Voice transmission means for transmitting voice information recorded in the other voice recording means to the one playback device;
The one playback device is:
Voice receiving means for receiving voice information transmitted from the voice transmitting means,
The plural audio recording means records the audio information received by the audio receiving means.

The invention described in claim 5
An image synchronization method for synchronously playing back an image related to the one scene using a plurality of playback devices that record a plurality of images obtained by capturing a scene with a plurality of imaging means,
Among the plurality of playback devices, at least one playback device is:
A plurality of sound recording means for recording a plurality of sound information recorded in association with a plurality of images obtained by capturing the one scene with a plurality of image capturing means;
Acquiring image synchronization information for reproducing a plurality of images synchronously by the plurality of reproduction devices based on the plurality of audio information recorded in the plurality of audio recording means;
Controlling the reproduction timing of the image based on the acquired image synchronization information;
Performing the step of transmitting the acquired image synchronization information to another playback device different from the one playback device among the plurality of playback devices;
The other playback device is
Receiving the transmitted image synchronization information;
And a step of controlling the reproduction timing of the image based on the received image synchronization information.

  According to the present invention, a synchronized reproduction of a plurality of images can be easily performed even if the images are not captured by a plurality of imaging means with accurate synchronization.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a diagram schematically illustrating a schematic configuration of an image synchronization system 100 according to an embodiment to which the present invention is applied, and FIG. 2 is a block diagram illustrating a schematic configuration of an imaging apparatus 1 configuring the image synchronization system 100. It is.

The image synchronization system 100 according to the present embodiment uses two images related to one scene captured by the two imaging devices (reproducing devices) 1 and 1 based on audio data recorded at the time of imaging the scene. Synchronized playback.
Specifically, the image synchronization system 100 includes, for example, two imaging devices 1 and 1 that capture an image of a subject, as shown in FIG.

  Each imaging device 1 has substantially the same configuration, one being a first imaging device (one imaging device) 1A operated by a first user and the other being operated by a second user. 2 imaging device (other imaging device) 1B. In addition, the image synchronization process is performed, for example, by the first imaging apparatus 1A.

Below, the structure of the imaging device 1 is demonstrated in detail with reference to FIG.
The imaging device 1 includes an imaging unit 2, a sound collection unit 3, an encoding unit 4, a memory 5, a decoding unit 6, a communication unit 7, an audio synchronization detection unit 8, an image reproduction unit 9, an audio reproduction unit 10, an operation unit 11, A control unit 12 and the like are provided.

  The imaging unit 2 is for imaging a subject. Specifically, for example, the imaging unit 2 includes an imaging lens, an electronic imaging unit, an AE / AF control unit, an A / D conversion unit, an image processing unit, and the like, although not illustrated.

  The imaging lens has, for example, a focus function and a zoom function, and includes a plurality of imaging lenses.

  The electronic imaging unit includes an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor) that converts a subject image that has passed through the imaging lens into a two-dimensional image signal.

  The A / D conversion unit includes, for example, a prefilter, an ADC (Analog Digital Converter) circuit, and the like, and converts an analog image signal output and input from the electronic imaging unit into a digital image signal.

The AE / AF control unit performs an automatic exposure adjustment (AE) process and an automatic focus adjustment (AF) process based on the digital image signal output from the A / D conversion unit and input.
The automatic exposure adjustment process is a process for automatically adjusting the exposure of the image sensor in accordance with the brightness of the subject to be photographed. The automatic focus adjustment process is a process for automatically adjusting the focus of the photographing lens with respect to the photographing target.

  The image processing unit performs predetermined image processing on the input image signal output from the AE / AF control unit and outputs the image signal to the control unit 12.

  The sound collection unit 3 collects and records sound, and includes a microphone, an A / D conversion unit, and the like.

The microphone collects sound when the subject is imaged.
The A / D conversion unit includes, for example, a prefilter, an ADC circuit, and the like, and converts an analog audio signal output from a microphone and input into a digital audio signal in PCM format.

  The encoding unit 4 includes, for example, an encoder, and compresses various data using a predetermined compression method. That is, the encoding unit 4 compresses digital image data of an image captured by the imaging unit 2 into a file format such as MPEG4, for example. Also, the encoding unit 4 compresses the PCM format digital audio data of the sound collected by the sound collection unit 3 into a file format such as MP3, for example. This is preferable in that the amount of data can be reduced and the transmission time of digital audio data by the communication unit 7 can be shortened.

The memory 5 is composed of, for example, a nonvolatile memory, a hard disk, etc., and records compressed digital image data and compressed digital audio data.
Here, the memory 5 records encoded image data of an image obtained by imaging one scene by the imaging unit 2 as an image recording unit. That is, the memory 5 of the first imaging device 1A records compressed digital image data of an image obtained by capturing one scene by the imaging unit 2 of the first imaging device 1A, while the second imaging device 1B The memory 5 records compressed digital image data of an image obtained by imaging one scene by the imaging unit 2 of the second imaging device 1B.

  The memory 5 records the compressed digital audio data in association with the encoded image data. That is, the memory 5 of the first imaging device 1A records the compressed digital audio data of the audio recorded by the sound collecting unit 3 in association with the image captured by the imaging unit 2 of the first imaging device 1A. On the other hand, the memory 5 of the second imaging device 1B is a voice recorded by the sound collection unit 3 in association with an image captured by the imaging unit 2 of the second imaging device 1B as another audio recording unit. Record compressed digital audio data.

In addition, the memory 5 of the first imaging device 1A stores the sound data collected by the sound collection unit 3 of the first imaging device 1A for calculation of time lag information (described later) by the audio synchronization detection unit 8. In addition, the audio data collected by the sound collection unit 3 of the second imaging device 1B is recorded. That is, the memory 5 of the first imaging device 1A records PCM digital audio data obtained by decoding the compressed digital audio data by the decoding unit 6. Further, the memory 5 is a PCM digital audio obtained by decoding the compressed digital audio data transmitted from the communication unit 7 of the second imaging device 1B and received by the communication unit 7 of the first imaging device 1A by the decoding unit 6. Record the data.
Here, the memory 5 of the first image pickup apparatus 1A constitutes a plurality of sound recording means for recording a plurality of sound data recorded in association with a plurality of images picked up by the plurality of image pickup units 2,. Yes.

  The decoding unit 6 includes a decoder, for example, and decodes various data compressed by a predetermined compression method. That is, the decoding unit 6 decodes data in a file format such as MPEG4 into digital image data, for example. The decoding unit 6 also decodes data in a file format such as MP3 into digital audio data in PCM format, for example.

The communication unit 7 transmits and receives various data to and from the communication unit 7 of the other imaging device 1 using, for example, infrared rays. Specifically, although not illustrated, the communication unit 7 includes a transmission unit that transmits various data by infrared rays and a reception unit that receives infrared rays transmitted from the communication units 7 of other imaging apparatuses.
That is, the communication unit 7 of the second imaging device 1B transmits the compressed digital audio data recorded in the memory 5 of the second imaging device 1B as an audio transmission unit (see FIG. 3).
Further, the first imaging device 1A receives the compressed digital audio data transmitted from the communication unit 7 of the second imaging device 1B as an audio receiving unit.

Further, the communication unit 7 of the first imaging device 1A transmits the time shift information acquired by the audio synchronization detection unit 8 to the second imaging device 1B as a synchronization information transmission unit (see FIG. 4).
On the other hand, the communication unit 7 of the second imaging device 1B receives the time shift information transmitted from the communication unit 7 of the first imaging device 1A as a synchronization information receiving unit.

  The audio synchronization detection unit 8 serves as image synchronization information acquisition means, based on digital audio data recorded at the time of imaging one scene by the two imaging devices 1, 1, and a time related to the deviation time between the audio data Get shift information. That is, the audio synchronization detection unit 8 is based on two PCM format digital audio data read from the memory 5, and the image reproduction unit 9 of the first imaging device 1A and the image reproduction unit of the second imaging device 1B. 9 calculates time shift information for reproducing the image synchronously.

Hereinafter, a method for calculating the time lag information will be described in detail with reference to FIGS. 5 (a) and 5 (b).
Here, FIG. 5A is a diagram schematically showing the correspondence between the audio data acquired by the first imaging device 1A and the captured image, and FIG. 5B is the second imaging. It is a figure which shows typically a response | compatibility with the audio | voice data acquired by the apparatus 1B, and the imaged image.
In FIG. 5A, the reproduced image at time t1 is image A1, the reproduced image at time t2 is image A2, and the reproduced image at time t3 is image A3. In FIG. 5B, the reproduced image at time t4 is referred to as an image B4, and the reproduced image at time t5 is referred to as an image B5.

The audio synchronization detection unit 8 indicates mutual similarity of time series waveforms based on the digital audio data of the first imaging device 1A and the digital audio data of the second imaging device 1B according to the following equation (1). Calculate the correlation function. Specifically, the digital audio data of the first imaging device 1A is x (t), the digital audio data of the second imaging device 1B is y (t), and the audio signal x ( t) is fixed, and the cross-correlation function Rxy (t) is calculated while increasing the time shift τ from −P to P within the predetermined search range N of the audio signal y (t) of the second imaging device 1B. .

The time shift τ at which the cross-correlation function Rxy (t) has the maximum value is set as shift information τ1 in the first search range N.
The cross-correlation function Rxy (t) is moved within the search range N by a predetermined size (for example, 2 / N) to obtain a time shift τ at which the cross-correlation function Rxy (t) takes a maximum value. Accordingly, for example, when the search range N is moved (k−1) times, k time differences are calculated as t1, τ2,.
Here, the audio synchronization detection unit 8 functions as a shift information search unit that searches for a plurality of shift information that takes the maximum value of the cross-correlation function Rxy (t) within a predetermined search range N of two digital audio data.

ここで、音声同期検出部８は、複数のずれ情報の平均値を算出して時間ずれ情報とする画像同期情報算出手段として機能する。 Next, the audio synchronization detection unit 8 calculates the average value Aveτ of the k pieces of deviation information according to the following formula (2), and the digital audio data x (t) of the first imaging device 1A and the second imaging A time lag τ between the digital audio data y (t) of the device 1B and the time is τ.

Here, the audio synchronization detection unit 8 functions as an image synchronization information calculation unit that calculates an average value of a plurality of pieces of shift information and sets it as time shift information.

  Specifically, for example, as shown in FIGS. 5A and 5B, the waveform portion w1 of the audio signal near the time t2 and the second imaging after the start of imaging by the first imaging device 1A. The degree of similarity with the waveform portion w2 of the audio signal in the vicinity of time t4 immediately after the start of imaging by the apparatus 1B is high. Thereby, the time from the start of imaging by the first imaging device 1A to the start time ta of the waveform portion w1 can be set as the time shift τ.

  The image reproduction unit 9 includes, for example, a DAC circuit, a display panel, and the like, converts a digital image signal into an analog image signal, and displays an image on the display panel. That is, the image reproduction unit 9 displays (reproduces) a predetermined image as image reproduction means based on the image data read from the memory 5 and decoded by the decoding unit 6 under the control of the control unit 12.

The sound reproducing unit 10 includes, for example, a DAC circuit, a speaker, and the like, converts a digital sound signal into an analog sound signal, and outputs sound from the speaker.
In the image synchronization process, only the audio reproduction unit 10 of the first imaging device 1A outputs audio.

  The operation unit 11 is for performing a predetermined operation of the imaging apparatus 1 and includes, for example, various operation buttons such as a shutter button and a mode switching button, which are not illustrated.

The shutter button instructs imaging of the subject by the imaging unit 2.
The mode switching button instructs switching between a still image / moving image capturing mode and an image reproduction mode.
When various operation buttons are operated, a predetermined operation signal is output to the CPU 121.

  The control unit 12 controls each unit of the imaging apparatus 1 and includes, for example, a CPU 121, a RAM 122, a ROM 123, and the like.

  The CPU 121 performs various control operations in accordance with various processing programs for the imaging apparatus 1 stored in the ROM 123.

  The RAM 122 includes a program storage area for expanding a processing program executed by the CPU 121, a data storage area for storing input data, a processing result generated when the processing program is executed, and the like.

  The ROM 123 stores various programs and data necessary for the operation of the CPU 121. Specifically, the ROM 123 stores a first reproduction control program 123a, a second reproduction control program 123b, and the like.

The first reproduction control program 123a causes the CPU 121 to function as first reproduction control means. That is, the first reproduction control program 123 a has a function related to the first reproduction control process for controlling the reproduction timing of the image by the image reproduction unit 9 based on the time shift information acquired by the audio synchronization detection unit 8. It is a program for realizing. Specifically, in the first imaging device 1A, based on the execution of the first reproduction control program 123a by the CPU 121, the image reproduction unit 9 generates an image according to the time lag information acquired by the audio synchronization detection unit 8. A first reproduction control process for controlling reproduction timing is executed.
In the first reproduction control process, for example, when the time lag τ is positive, that is, the imaging of the second imaging device 1B is started with a predetermined time delay after the imaging of the first imaging device 1A is started. In this case, when the image reproduction of the first imaging device 1A is instructed, the CPU 121 starts the reproduction of the image by the image reproduction unit 9, and the reproduction start signal of the second imaging device 1B and a predetermined value. The time shift information related to the time shift τ is transmitted from the communication unit 7.
Further, for example, when the time shift τ is negative, that is, when imaging of the first imaging device 1A is started with a predetermined time delay after imaging of the second imaging device 1B is started, the first imaging is performed. When the image reproduction of the apparatus 1A is instructed, the CPU 121 causes the communication unit 7 to transmit the reproduction start signal of the second imaging apparatus 1B and the time deviation information of time deviation τ = 0, and only the time deviation τ. Start playing an image with a delay.

The second reproduction control program 123b causes the CPU 121 to function as second reproduction control means. In other words, the second playback control program 123b provides the CPU 121 with a function related to the second playback control process for controlling the playback timing of the image by the image playback section 9 based on the time shift information received via the communication section 7. It is a program for realizing. Specifically, in the second imaging device 1B, based on the execution of the second reproduction control program 123b by the CPU 121, according to the time lag information transmitted from the first imaging device 1A and received by the communication unit 7. A second reproduction control process for controlling the reproduction timing of the image by the image reproduction unit 9 is executed.
In the second reproduction control process, for example, when time lag information having a positive time lag τ is received, the CPU 121 delays the image by the time lag τ after receiving the reproduction start signal. Is started (see FIG. 6).
Further, for example, when the time shift information of time shift τ = 0 is received, the CPU 121 starts playback of the image immediately after receiving the playback start signal.

Next, image synchronization processing will be described with reference to FIGS.
Here, FIG. 6 is a flowchart showing an example of the operation related to the image synchronization processing, and FIG. 7 shows the correspondence between the images reproduced by the first imaging device 1A and the second imaging device 1B and the audio data. It is a figure shown typically.

  As shown in FIG. 6, the first user uses the first imaging device 1A, and the second user uses the second imaging device 1B to capture a moving image of one scene from different directions ( Step S1 and Step S2). At this time, the first user and the second user do not need to accurately match the imaging start time, and the imaging is started individually so that a desired scene is recorded.

  In the second imaging device 1B, when the imaging by the imaging unit 2 is completed (step S3), the CPU 121 causes the communication unit 7 to transmit the compressed digital audio data recorded in the memory 5 (step S4).

  On the other hand, in the first imaging apparatus 1A, when the imaging by the imaging unit 2 is completed (step S5), the CPU 121 decodes the compressed digital audio data by the decoding unit 6 and stores the digital audio data in the PCM format in the memory 5. (Step S6). Further, when the CPU 121 receives the compressed digital audio data transmitted from the second imaging device 1B by the communication unit 7 (step S7), the CPU 121 decodes the data by the decoding unit 6 and stores the digital audio data in the PCM format in the memory 5. (Step S8).

  Next, the CPU 121 of the first imaging apparatus 1A controls the audio synchronization detection unit 8 to calculate time shift information based on the two PCM format digital audio data read from the memory 5 (step S1). S9).

  Thereafter, in a state where the reproduction mode of the first imaging device 1A and the second imaging device 1B is set to the image synchronous reproduction mode, the image synchronous reproduction is performed based on a predetermined operation of the operation unit 11 of the first imaging device 1A. When execution is instructed (step S10), the CPU 121 controls the communication unit 7 to transmit a reproduction start signal of the second imaging device 1B and time shift information related to a predetermined time shift τ (step S11). .

Thereafter, the CPU 121 of the first imaging device 1A executes a predetermined determination program (not shown) read from the ROM 123 to determine whether or not the time lag τ is positive (step S12). Here, if it is determined that the time lag τ is positive (step S12; YES), the CPU 121 reads out and executes the first reproduction control program 123a from the ROM 123, and the image reproduction unit 9 reproduces the image. Start (step S13).
On the other hand, in the second imaging device 1B, when the reproduction start signal and the time shift information transmitted from the first imaging device 1A are received by the communication unit 7 (step S14), the CPU 121 reads the predetermined read from the ROM 123. This determination program (not shown) is executed to determine whether or not the time lag τ is negative (step S15). Here, if it is determined that the time lag τ is not negative (step S15; NO), the CPU 121 reads and executes the second reproduction control program 123b from the ROM 123 and executes the time after receiving the reproduction start signal. After waiting for the time of deviation τ (step S16), the image reproduction unit 9 starts image reproduction (step S17).
Thereby, for example, as shown in FIG. 7, after the image is reproduced by the first imaging device 1A, the image is reproduced only by the first imaging device 1A until time ta. After time ta, the image A2 is displayed on the image reproduction unit 9 of the first imaging device 1A at time t2 (t4), and the image B4 is displayed on the image reproduction unit 9 of the second imaging device 1B. At the time t3 (t5), the image A3 is displayed on the image reproduction unit 9 of the first imaging device 1A, and the image B5 is displayed on the image reproduction unit 9 of the second imaging device 1B. In addition, the two imaging devices 1 and 1 reproduce images in synchronization.

In the first imaging apparatus 1A, when it is determined in step S12 that the time lag τ is not positive (step S12; NO), the CPU 121 loads the first reproduction control program 123a from the ROM 123. After reading and executing and waiting for the time difference τ (step S18), the process proceeds to step S13, and the image reproduction unit 9 starts image reproduction.
On the other hand, in the second imaging device 1B, when it is determined in step S15 that the time lag τ is negative (step S15; YES), the CPU 121 loads the second reproduction control program 123b from the ROM 123. Read and execute, proceed to step S17, and immediately start image reproduction by the image reproduction unit 9.

  As described above, according to the image synchronization system 100 of the present embodiment, the second imaging device 1B transmits the audio data recorded at the time of imaging one scene to the first imaging device 1A. The first imaging device 1A can calculate time lag information based on two audio data recorded during imaging of one scene by the first imaging device 1A and the second imaging device 1B. Based on the time shift information, the reproduction timing of the image recorded in the first imaging device 1A can be controlled. Furthermore, the second imaging device 1B can control the reproduction timing of the image recorded in the second imaging device 1B based on the time shift information transmitted from the first imaging device 1A. Therefore, even if the images are not captured by the two imaging devices 1 and 1 with accurate synchronization, the two images can be easily synchronized and reproduced based on the time shift information.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
Hereinafter, modified examples of the image synchronization system 100 will be described with reference to FIGS.

<Modification 1>
As shown in FIG. 8, the audio synchronization detection unit 208 of the imaging apparatus 201 constituting the image synchronization system of Modification 1 calculates an average value Aveτ of a plurality of time shifts τ using the maximum value of the cross-correlation function itself as a weighting coefficient. To do.
Specifically, the maximum value Rxy (τm) of the cross-correlation function Rxy (τ) in the m-th search range obtained by the equation (1) is stored in the memory 5, and the Rxy (τm) is time-shifted. Let τ be the reliability. That is, as the cross-correlation function Rxy (τm) is higher, the reliability of the corresponding τm is higher. Therefore, the average value Aveτ is calculated by the equation (3) using the cross-correlation function Rxy (τm) as a weighting coefficient.
When the maximum value Rxy (τm) is not stored in the memory 5, the calculation may be performed again by substituting τm for τ in the equation (1).

Therefore, according to the image synchronization system of the first modification, the digital audio data x (t) of the first imaging device and the digital audio data y (t) of the second imaging device are mutually within a predetermined search range N. When calculating the average value Aveτ of a plurality of deviation information taking the maximum value of the correlation function Rxy (τ) to obtain time deviation information, the average value Aveτ is calculated using the maximum value Rxy (τm) of the cross-correlation function as a weighting coefficient. By doing so, more reliable time shift information can be calculated.
In other words, when recording the same sports scene or motion of the form at the same time, in addition to the sound that is the purpose of calculating the time lag information such as the hitting sound, swing sound, and subject sound generated by the sports scene, the surrounding environment Noise is also collected. At this time, if the target sound is generated in an almost quiet environment, the deviation information with the search range of the speech section of the noise part that is random and has no periodicity has an accurate time delay. It cannot be said that it represents. Therefore, by using the maximum value Rxy (τm) of the cross-correlation function as a weighting coefficient, it is possible to calculate more reliable time shift information.

<Modification 2>
In the image synchronization system of the second modification, among the two moving images (image groups) recorded in the memory 5 of the two image pickup devices 301 and 301, the two image pickup devices 301 and 301 have taken images in duplicate. Only duplicate images are played back.

  That is, in the ROM 123 of the control unit 312 of the imaging device 301 that constitutes the image synchronization system, for example, as shown in FIG. 9, in addition to the first reproduction control program 123a and the second reproduction control program 123b, priority is given. An information acquisition program 123c and an end information acquisition program 123d are recorded.

The priority information acquisition program 123c causes the CPU 121 to function as priority information acquisition means. In other words, the priority information acquisition program 123c, based on the time shift information acquired by the audio synchronization detection unit 8, of the two moving images captured by the two imaging devices 301, the moving image whose imaging has started later. The CPU 121 implements a function related to the priority information acquisition processing for acquiring the priority moving image specifying information (priority image group specifying information) that is used as the priority moving image (priority image group) for starting playback from the head image by the image playback unit 9. It is a program for.
Specifically, based on the execution of the priority information acquisition program 123c by the CPU 121 of the first imaging device, when the time lag τ is positive, that is, after the start of imaging of the first imaging device, the second delays after a predetermined time. When imaging of the imaging device is started, the priority moving image specifying information is acquired using the moving image captured by the second imaging device as the priority moving image, while the time shift τ is negative, That is, when imaging of the first imaging device is started with a predetermined time delay after the start of imaging by the second imaging device, the priority moving image that uses the moving image captured by the first imaging device as the priority moving image. Process to acquire specific information.

The end information acquisition program 123d causes the CPU 121 to function as end information acquisition means. That is, the end information acquisition program 123d is a moving image whose reproduction ends early after the start of the reproduction of the priority moving image, out of two moving images that are reproduced in synchronization by the image reproducing unit based on the two audio data. Is a program for causing the CPU 121 to realize a function related to an end information acquisition process for acquiring early end moving image specifying information (early end image group specifying information).
Specifically, based on the execution of the end information acquisition program 123d by the CPU 121, each audio signal is based on the waveform portion w1 (w2) having a high similarity in the two audio signals detected by the audio synchronization detection unit 8. The moving image whose reproduction ends earlier than the two moving images by acquiring the length from the waveform portion w1 (w2) to the end of the sound and comparing them with each other is defined as the early ending moving image. A process of acquiring early end moving image specifying information is performed.

In addition, the communication unit 307 of the first imaging device uses the priority moving image specifying information acquired in the priority information acquiring process and the early end moving image specifying information acquired in the end information acquiring process as the specific information transmitting unit. Transmit to the second imaging device.
On the other hand, the communication unit 307 of the second imaging device receives the priority moving image specifying information and the early termination moving image specifying information transmitted from the communication unit 307 of the first imaging device as the specific information receiving unit.

Next, an image synchronization process performed by the image synchronization system according to the second modification will be described.
In the image synchronization process, the first imaging device reproduces the overlapping timing of the moving image reproduction timing by the image reproducing unit 9 based on the priority moving image specifying information and the early end moving image specifying information. A first reproduction control process to be controlled is performed. In addition, the second imaging device is configured to reproduce the moving image by the image reproducing unit 9 based on the priority moving image specifying information and the early end moving image specifying information transmitted from the first imaging device and received via the communication unit. The second reproduction control process is performed to control the reproduction so that the overlapping images are reproduced in synchronization.
Specifically, when the captured moving image is a priority moving image, the first imaging device and the second imaging device perform reproduction from the top image of the priority moving image based on the priority moving image specifying information. On the other hand, if it is not a priority moving image, reproduction is performed from an image captured at the same time as the head image of the priority moving image in the non-priority moving image. In addition, one of the first imaging device and the second imaging device is based on the early end moving image specifying information when the moving image captured by the imaging device is not the early end moving image. The reproduction of the non-early end moving image is stopped in accordance with the end timing of the early end moving image reproduced by the other imaging device.

Hereinafter, a synchronized reproduction mode in the image synchronization process will be described with reference to FIGS. 10 (a) to 10 (d).
Here, FIG. 10A is an example in which an image captured by the second imaging device is used as a priority moving image and an early end moving image, and FIG. 10B is an example of the second imaging device. FIG. 10C illustrates an example in which the captured image is set as the priority moving image and the image captured by the first imaging device is the early end moving image. FIG. 10C illustrates an image captured by the first imaging device. 10D is an example in which the priority moving image is an early end moving image. FIG. 10D illustrates the image captured by the first imaging device as the priority moving image and the image captured by the second imaging device. This is an example of an early end moving image.
In FIGS. 10A to 10D, “α” represents the head image of the priority moving image, and “β” is captured at the same time as the head image α of the priority moving image in the non-priority moving image. Represents the image. In addition, a portion indicated by a solid line in the moving image represents an overlapping image.

In the synchronous playback mode shown in FIG. 10A, the CPU 121 of the first imaging device controls the image playback unit 9 based on the priority moving image specifying information at the same time as the first image α of the priority moving image. Playback is started from the imaged image β, and in synchronization with this, the CPU 121 of the second imaging device controls the image playback unit 9 based on the priority moving image specifying information to start from the first image of the priority moving image. Perform the playback process.
Further, the CPU 121 of the second imaging device controls the image reproduction unit 9 based on the early end moving image specifying information to stop the reproduction of the moving image (early end moving image), and in synchronization with this, The CPU 121 of the first imaging apparatus performs a process of stopping the reproduction of the moving image in accordance with the end timing of the early end moving image based on the early end moving image specifying information.

In the case of the synchronous playback mode shown in FIG. 10B, the CPU 121 of the first imaging device controls the image playback unit 9 based on the priority moving image specifying information at the same time as the first image α of the priority moving image. Playback is started from the imaged image β, and in synchronization with this, the CPU 121 of the second imaging device controls the image playback unit 9 based on the priority moving image specifying information to start from the first image of the priority moving image. Perform the playback process.
Further, the CPU 121 of the first imaging device controls the image playback unit 9 based on the early end moving image specifying information to stop the playback of the moving image (early end moving image), and in synchronization with this, The CPU 121 of the second imaging apparatus performs a process of stopping the reproduction of the moving image in accordance with the end timing of the early end moving image based on the early end moving image specifying information.

In the case of the synchronous playback mode shown in FIG. 10C, the CPU 121 of the first imaging device controls the image playback unit 9 based on the priority moving image specifying information to play back from the first image of the priority moving image. The CPU 121 of the second imaging device controls the image reproduction unit 9 based on the priority moving image specifying information and reproduces from the image β captured at the same time as the first image α of the priority moving image. Process to start.
Further, the CPU 121 of the first imaging device controls the image playback unit 9 based on the early end moving image specifying information to stop the playback of the moving image (early end moving image), and in synchronization with this, The CPU 121 of the second imaging apparatus performs a process of stopping the reproduction of the moving image in accordance with the end timing of the early end moving image based on the early end moving image specifying information.

In the case of the synchronous playback mode shown in FIG. 10D, the CPU 121 of the first imaging device controls the image playback unit 9 based on the priority moving image specifying information to play back from the first image of the priority moving image. The CPU 121 of the second imaging device controls the image reproduction unit 9 based on the priority moving image specifying information and reproduces from the image β captured at the same time as the first image α of the priority moving image. Process to start.
Further, the CPU 121 of the second imaging device controls the image reproduction unit 9 based on the early end moving image specifying information to stop the reproduction of the moving image (early end moving image), and in synchronization with this, The CPU 121 of the first imaging apparatus performs a process of stopping the reproduction of the moving image in accordance with the end timing of the early end moving image based on the early end moving image specifying information.

  Therefore, according to the image synchronization system of the second modification, out of the moving images captured by the two imaging devices 301 and 301, non-priority is matched with the reproduction timing of the first image of the priority moving image that has been started to be captured later. The reproduction of the moving image can be started, and the reproduction of the non-early end moving image can be stopped in accordance with the stop timing of the early end moving image whose reproduction ends earlier after the start of the reproduction of the priority image group. As a result, it is possible to properly reproduce only the overlapped image captured by the two image pickup devices 301 and 301, and to provide a more convenient image synchronization system.

  In the above embodiment, infrared communication is exemplified as the communication method of the communication unit 7, but is not limited to this, for example, a wireless LAN conforming to the Bluetooth (registered trademark) standard, the IEEE 802.11 standard, or the like, A wired system using a wired LAN, a USB cable, or the like may be used.

  In the above embodiment, the image synchronization processing is performed based on the image data and audio data recorded by the first imaging device 1A. However, the present invention is not limited to this. For example, another imaging device 1 The image data and audio data recorded by the above may be acquired via the communication unit 7 and image synchronization processing may be performed based on these data. At this time, the display panel of the image reproduction unit 9 of the first imaging device 1A may be a multi-screen and a plurality of images may be displayed simultaneously.

Furthermore, in the above-described embodiment, the image synchronization system 100 is illustrated as having the configuration including the two imaging devices 1 and 1, but the number of the imaging devices 1 is not limited to two, and may be any plural number.
In this case, by setting an ID number for each imaging device 1 and transmitting the ID number to the time lag information or the reproduction start signal, the predetermined time lag information or the like is accurately transmitted to each imaging device 1. The image synchronization processing can be reliably executed with a simple configuration.

  The second imaging device 1B may be configured to operate in conjunction with the operation of the first imaging device 1A. For example, operations such as playback, fast forward, rewind, and slow playback in the first imaging device 1A. Based on the above, the operation of the second imaging device 1B may be linked.

  In addition, in the above-described embodiment, the functions as the first reproduction control unit and the second reproduction control unit are realized by the CPU 121 executing a predetermined program or the like. For example, you may comprise from the logic circuit etc. for implement | achieving various functions.

It is a figure which shows typically schematic structure of the image synchronization system of one Embodiment to which this invention is applied. It is a block diagram which shows schematic structure of the imaging device which comprises the image synchronization system of FIG. It is a figure which shows typically the communication state of the imaging devices in the image synchronization process by the image synchronization system of FIG. It is a figure which shows typically the communication state of the imaging devices in the image synchronization process by the image synchronization system of FIG. It is a figure which shows typically the response | compatibility with the audio | voice data acquired by the imaging device which comprises the image synchronization system of FIG. 1, and the imaged image. 3 is a flowchart illustrating an example of an operation related to image synchronization processing by the image synchronization system of FIG. 1. It is a figure which shows typically a response | compatibility with the image reproduced | regenerated by the imaging device which comprises the image synchronization system of FIG. 1, and audio | voice data. It is a block diagram which shows schematic structure of the imaging device which comprises the image synchronization system of the modification 1. It is a block diagram which shows schematic structure of the imaging device which comprises the image synchronization system of the modification 2. It is a figure for demonstrating an example of the synchronous reproduction | regeneration aspect in the image synchronization process by the image synchronization system of FIG.

Explanation of symbols

100 Image synchronization system 1, 201, 301 Imaging device (reproducing device)
2 Imaging unit (imaging means)
311 Operation unit (operation input means)
5 Memory (image recording means, multiple audio recording means, other audio recording means)
7 Communication unit (synchronization information transmission means, synchronization information reception means, voice transmission means, voice reception means)
8, 208 Audio synchronization detection unit (image synchronization information acquisition means, deviation information search means, image synchronization information calculation means)
9,309 Image reproduction unit (image reproduction means, imaging order notification means)
121 CPU (first reproduction control means, second reproduction control means)

Claims (5)

An image synchronization system comprising a plurality of playback devices that play back images related to one scene in synchronization,
Each of the plurality of playback devices is
Image recording means for recording any one of a plurality of images obtained by capturing the one scene with a plurality of imaging means;
Image reproducing means for reproducing an image recorded in the image recording means,
Among the plurality of playback devices, at least one playback device is:
A plurality of sound recording means for recording a plurality of sound information recorded in association with a plurality of images picked up by the plurality of image pickup means;
Image synchronization information acquisition for acquiring image synchronization information for synchronizing and reproducing a plurality of images by the image reproduction means of the plurality of reproduction devices based on the plurality of audio information recorded in the plurality of audio recording means Means,
First reproduction control means for controlling the reproduction timing of the image by the image reproduction means based on the image synchronization information acquired by the image synchronization information acquisition means;
Synchronization information transmitting means for transmitting the image synchronization information acquired by the image synchronization information acquiring means to another playback device different from the one playback device among the plurality of playback devices,
The other playback device is:
Synchronization information receiving means for receiving the image synchronization information transmitted from the synchronization information transmitting means;
An image synchronization system comprising: second reproduction control means for controlling reproduction timing of an image by the image reproduction means based on the image synchronization information received by the synchronization information receiving means. The image synchronization information acquisition means includes
Deviation information search means for searching for a plurality of deviation information taking a maximum value of a cross-correlation function within a predetermined search range of the plurality of voice information;
Image synchronization information calculating means for calculating an average value of the plurality of deviation information searched by the deviation information searching means and making the image synchronization information,
The image synchronization system according to claim 1, wherein the image synchronization information calculation unit calculates the average value using a maximum value of the cross-correlation function as a weighting coefficient. The one playback device is:
Based on the image synchronization information acquired by the image synchronization information acquisition means, an image group whose imaging has been started later is selected by the image reproduction means from among the plurality of image groups captured by the plurality of imaging means. Priority information acquisition means for acquiring priority image group identification information as a priority image group for starting playback from:
An image group whose reproduction ends early after the reproduction of the priority image group by the image reproduction unit among the plurality of image groups reproduced in synchronization by the image reproduction unit based on the plurality of audio information An end information acquisition means for acquiring early end image group identification information,
Specific information transmitting means for transmitting the priority image group specifying information acquired by the priority information acquiring means and the early end image group specifying information acquired by the end information acquiring means to the other playback device,
The first reproduction control unit is further configured to reproduce the image based on the priority image group specifying information acquired by the priority information acquiring unit and the early end image group specifying information acquired by the end information acquiring unit. Controlling the reproduction timing of the image group by the means so as to reproduce the duplicate images captured by the plurality of imaging means in a synchronized manner,
The other playback device is:
Specific information receiving means for receiving the priority image group specifying information and the early ending image group specifying information transmitted from the specific information transmitting means,
The second reproduction control unit further sets the reproduction timing of the image by the image reproduction unit based on the priority image group identification information and the early end image group identification information received by the identification information reception unit. The image synchronization system according to claim 1, wherein the image synchronization system is controlled so as to reproduce the image in synchronization. The other playback device is:
Other audio recording means for recording audio information recorded in association with the one image recorded in the image recording means;
Voice transmission means for transmitting voice information recorded in the other voice recording means to the one playback device;
The one playback device is:
Voice receiving means for receiving voice information transmitted from the voice transmitting means,
The image synchronization system according to any one of claims 1 to 3, wherein the plurality of sound recording means records sound information received by the sound receiving means. An image synchronization method for synchronously playing back an image related to the one scene using a plurality of playback devices that record a plurality of images obtained by capturing a scene with a plurality of imaging means,
Among the plurality of playback devices, at least one playback device is:
A plurality of sound recording means for recording a plurality of sound information recorded in association with a plurality of images obtained by capturing the one scene with a plurality of image capturing means;
Acquiring image synchronization information for reproducing a plurality of images synchronously by the plurality of reproduction devices based on the plurality of audio information recorded in the plurality of audio recording means;
Controlling the reproduction timing of the image based on the acquired image synchronization information;
Performing the step of transmitting the acquired image synchronization information to another playback device different from the one playback device among the plurality of playback devices;
The other playback device is
Receiving the transmitted image synchronization information;
And a step of controlling the reproduction timing of the image based on the received image synchronization information.

Images