JP6983364B1 - Video synchronization device, video synchronization method, and video synchronization program - Google Patents

Abstract

The video synchronization device (200) includes a feature point extraction unit (210), a motion amount calculation unit (220), and a time correction unit (240). The feature point extraction unit (210) is a feature of at least one of the feature points in each of a plurality of images of a subject including a feature point indicating a position where the coordinates change with the movement of the subject from a plurality of different viewpoints. Extract points. The movement amount calculation unit (220) obtains at least one movement amount indicating the amount of movement of at least one feature point in each of the plurality of images. The time correction unit (240) obtains a time correction amount used to synchronize a plurality of images based on at least one movement amount in each of the plurality of images.

Description

The present disclosure relates to a video synchronization device, a video synchronization method, and a video synchronization program.

When shooting a person, multiple cameras may be used at the same time to shoot the person in order to eliminate blind spots. In order to eliminate the time lag between multiple images shot by multiple cameras, it is necessary to synchronize multiple images.

International Publication No. 2013/094115

Patent Document 1 discloses a device that synchronizes a plurality of images by measuring the amount of time lag of the images by analyzing the contents of each of the plurality of images.
In Patent Document 1, the person state is sequentially estimated from each camera image, and the time lag amount is measured and corrected by adjusting the timing at which the person state changes. Here, the person state is a state related to the person of interest and refers to a "sitting state", an "upright state", or the like. As a specific example, when synchronizing multiple images of a person standing up from a chair, specify the frame that transitions from the "sitting state" to the "upright state" for each image so that there is no time difference between the specified frames. Correct the amount of time lag.
However, since the device disclosed in Patent Document 1 uses a state recognizer that recognizes a person's state, there is a problem that it cannot be applied to an image with a person's state that cannot be recognized by the state recognizer. be.

An object of the present disclosure is to synchronize a plurality of images taken from a plurality of different viewpoints without using a state recognizer.

The video synchronization device according to the present disclosure is
A video synchronization device that synchronizes a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, a feature point extraction unit for extracting at least one feature point among the feature points, and a feature point extraction unit.
A motion amount calculation unit for obtaining at least one motion amount indicating the amount of motion of each of the at least one feature point in each of the plurality of images.
Each of the plurality of images includes a time correction unit for obtaining a time correction amount used for synchronizing the plurality of images based on the at least one motion amount.

According to the present disclosure, it is possible to synchronize a plurality of images taken from a plurality of different viewpoints without using a state recognizer.

A configuration example of the video synchronization system 90 according to the first embodiment. It is a figure explaining the operation example of the feature point extraction unit 210 which concerns on Embodiment 1, (a) is a specific example which concerns on a reference image, (b) is a specific example which concerns on the correction target image. The figure explaining the operation example of the movement amount calculation unit 220 which concerns on Embodiment 1. FIG. A configuration example of the time correction unit 240 according to the first embodiment. A hardware configuration example of the video synchronization device 200 according to the first embodiment. The flowchart which shows the operation of the image synchronization apparatus 200 which concerns on Embodiment 1. The flowchart which shows the operation of the time correction unit 240 which concerns on Embodiment 1. A hardware configuration example of the video synchronization device 200 according to the modified example of the first embodiment.

In the description and drawings of the embodiments, the same elements and corresponding elements are designated by the same reference numerals. The description of the elements with the same reference numerals will be omitted or simplified as appropriate. The arrows in the figure mainly indicate the flow of data or the flow of processing. Further, "part" may be appropriately read as "means", "circuit", "process", "procedure", "processing" or "circuit Lee".

Embodiment 1.
Hereinafter, the present embodiment will be described in detail with reference to the drawings.

*** Explanation of configuration ***
FIG. 1 shows a configuration example of the video synchronization system 90 according to the present embodiment. As shown in this figure, the video synchronization system 90 includes a video acquisition means 100, a video synchronization device 200, and a video output means 300.
In the video synchronization system 90, the video synchronization device 200 receives a plurality of videos asynchronously shot from the video acquisition means 100, and sends out a plurality of synchronized videos to the video output means 300. At least two of the video acquisition means 100, the video synchronization device 200, and the video output means 300 may be integrally configured. The image contains information showing how the subject is moving. The video may be read as a moving image.

The video synchronization system 90 can synchronize a plurality of videos. However, for convenience of explanation, unless otherwise specified, an example in which the video synchronization system 90 synchronizes two videos will be described below. The video synchronization system 90 synchronizes a plurality of videos by correcting the shooting time.
Typically, each of the plurality of images contains information indicating the shooting time, and the plurality of images are shot asynchronously from a plurality of different viewpoints. Multiple viewpoints are realized by a plurality of cameras installed at a plurality of different points. Typically, each of the plurality of cameras is fixed at each of the plurality of points, and the shooting range of each of the plurality of images is constant.
Each of the plurality of images has an image in a certain shooting period. The shooting period of each of the plurality of images may be different. At least a part of the target subject may not always be captured during the shooting period of each of the plurality of images. The target subject refers to a target for which the video synchronization device 200 detects movement. That is, the target subject is not always stationary during the shooting period of each of the plurality of images. The number of target subjects may be two or more. Hereinafter, the target subject may be simply referred to as a subject. The subject is, as a specific example, a person, an animal, a robot, a moving body, or a combination thereof.

There are at least two image acquisition means 100, and each of the image acquisition means 100 acquires an image including a subject. That is, the video acquisition means 100 acquires at least two videos. The image acquisition means 100 may acquire an image from a camera, or may acquire an image from a server or the like that stores the image. Hereinafter, it is assumed that the video acquisition means 100 acquires two videos. Of the two images, one is a reference image that serves as a reference for time correction, and the other is a correction target image that corrects the time. When the video synchronization system 90 synchronizes a plurality of videos, any one of the plurality of videos is the reference video, and the other video is the correction target video.
Further, in FIG. 1, regarding the video acquisition means 100, in order to distinguish between the means for acquiring the reference video and the means for acquiring the correction target video, the means for acquiring the reference video is referred to as the video acquisition means 100-1. , The means for acquiring the image to be corrected is referred to as the image acquisition means 100-2. Similarly, for each part of the video synchronization apparatus 200, "-1" is appropriately added to the part for processing the reference video, and "-2" is given to the part for processing the correction target video.

As shown in this figure, the video synchronization device 200 includes a feature point extraction unit 210, a motion amount calculation unit 220, a video buffer unit 230, a time correction unit 240, a correction amount storage unit 250, and a video synchronization unit 260. And prepare. The time correction unit 240 is also called a time correction amount calculation means. The correction amount storage unit 250 is also called a time correction amount storage unit. The image synchronization device 200 synchronizes a plurality of images of a subject including a feature point taken from a plurality of different viewpoints.

The feature point extraction unit 210 extracts at least one feature point among the feature points included in the subject in each of the plurality of images. The feature point indicates a position where the coordinates change with the movement of the subject. The feature point may point to a certain area, may point to a point shown in the image, or may indirectly point to a point not shown in the image. The subject contains at least one feature point. That is, the subject has a position corresponding to at least one feature point. As a specific example, the feature point extraction unit 210 extracts points satisfying a certain property as feature points for each frame of a moving image.
The characteristic point is, as a specific example, the position of a joint such as a person's shoulder, elbow, or wrist. The feature point extraction unit 210 may extract the joint position using a CNN (Convolutional Neural Network) trained to infer the joint position from the image of the person, and may use the position of the marker worn by the person as the joint position. It may be extracted as. Further, the feature point extraction unit 210 does not need to include the joint position in the feature point, and may extract the center position of the person image extracted by the foreground extraction process or the like as the feature point.

FIG. 2 is a diagram illustrating an operation example of the feature point extraction unit 210. This figure shows a specific example in which the feature point extraction unit 210 extracts the joint positions of the shoulder, elbow, wrist, etc. of the person as feature points when the subject is a person. (A) is an image showing the right side surface of the person, and shows an image at a certain point in time of the reference image. (B) is an image showing the left side surface of the person, and shows an image of the image to be corrected at a certain point in time. In this figure, the image taken out from the reference image or the image to be corrected is shown on the left side, and the image on the left side with the feature points added by circles is shown on the right side. In this example, (a) reflects the blind spot of (b), and (b) reflects the blind spot of (a).
M indicates the maximum number of feature points that can be extracted by the feature point extraction unit 210. That is, M is the maximum number of feature points for each subject, and is a constant common to the reference image and the correction target image. m indicates a feature point number. That is, m is a variable that identifies a feature point, and is an integer of 1 or more and M or less. The feature points p1 _{and m} mean that the m-th feature point among the feature points extracted from the reference image is indicated. The feature points p2 _{and m} mean that the m-th feature point among the feature points extracted from the corrected image is indicated. That is, in this figure, the feature points p _{1, m} and the feature points p _{2, m} are different in position in the image, but are the same feature points in the target subject, and both are the same m. Pointing to the second joint. Here, the relationship between the reference image and the correction target image can be defined by using the feature points extracted by the feature point extraction unit 210. When the subject indicated by the reference image and the subject indicated by the correction target image are the same subject, the feature points extracted by the feature point extraction unit 210 are common between the reference image and the correction target image. When the subject indicated by the reference image and the subject indicated by the correction target image are different, the feature points extracted by the feature point extraction unit 210 differ between the reference image and the correction target image. Also, unless the feature points of the subject are in the part where only a part of the subject is shown in the image and not shown in the image, the feature points indirectly indicate the points not shown in the image. The feature point numbers of the extracted feature points are not continuous but discrete values.

The movement amount calculation unit 220 obtains at least one movement amount indicating the amount of movement of at least one feature point in each of the plurality of images. Each of the at least one movement amount indicates the amount of movement of each of the at least one feature point. The amount of movement indicates the amount of movement of at least a part of the subject per unit time. The movement amount calculation unit 220 may obtain, for each of at least one movement amount, the amount by which the coordinate values of each of the at least one feature point change in a unit time. The amount of motion may be a physical quantity whose increase or decrease in the value of the amount of motion does not depend on the viewpoint of the image between the reference image to be compared and the image to be corrected. The physical quantity does not have to be observed at all time points. That is, the amount of movement is a physical quantity in which the same increase or decrease can be observed between a plurality of images. Here, two specific examples are shown when the increase / decrease of the physical quantity depends on the viewpoint of the image. The first is that when at least a part of the subject is moving, there is no change in the physical quantity indicating the movement of the moving part in at least one image, that is, the movement is observed in at least one image. Is something that cannot be done. The second is that when at least a part of the subject is moving, the increase / decrease of the physical quantity indicating the movement of the moving part is not uniform in all the images. That is, as a specific example, the physical quantity showing the movement tends to increase in one video, but the physical quantity showing the movement tends to decrease in the other video. As a specific example, the amount of movement is the amount of change in the coordinate value indicating the feature point per unit time.

FIG. 3 is a diagram illustrating an operation example of the motion amount calculation unit 220, and is a diagram corresponding to FIG. 2. In this figure, _{pn, m, t} indicate the coordinates of the feature points _{pn, m at time t.} The smaller the value of t, the earlier the time. When the values of t at the two feature points are continuous, the time interval between the two images including each of the two feature points is constant. n is a variable that identifies the image, and is 1 or 2.
FIG. 3 shows how the feature points indicating the wrists of a person move when the person waves one hand. In this figure, a plurality of frames are extracted by extracting frames from a reference video at unit time intervals, feature points in each of the plurality of frames are extracted, and the extracted feature points are superimposed and displayed.

The temporal distribution of the feature points shown in the center of FIG. 3 is uneven depending on the speed of movement of the person. The temporal distribution of feature points indicates the position of feature points at each time in a discrete-time system. As a specific example, the amount of movement is a physical quantity that expresses the degree of density of the temporal distribution of feature points. As a specific example, the movement amount calculation unit 220 determines the movement amount so that the movement speed of the person becomes a relatively large value when the movement speed is relatively fast, and becomes a relatively small value when the movement speed of the person is relatively slow. demand.
[Equation 1] shows a specific example of a calculation method in which the movement amount calculation unit 220 obtains the movement amount. [Equation 2] specifically indicates a part of [Equation 1]. The symbols used in [Equation 1] and [Equation 2] are the same as those in FIGS. 2 and 3.
[Equation 1] is a mathematical formula for obtaining a value similar to variance. Therefore, when the movement amount calculation unit 220 obtains the movement amount using [Equation 1], the sparser the temporal distribution of the feature points, the larger the movement amount becomes.
var _{n, m, t} indicates the amount of movement of _{pn, m, t.} In the view shown in the middle of the _{figure, p n, m, t 'and} the center, _{var n, m, t'} by showing a circle of the radius, shows the magnitude of the _{var n, m, t '} There is. Here, var _{n, m,} t'is the amount of movement of _{pn, m,} t'obtained by [Equation 1].
The graph shown on the right of this figure is a graph of the temporal distribution of feature points, and shows the time transition of the amount of movement. The horizontal axis shows the time, and the vertical axis shows the amount of movement. In this graph, the points corresponding to the feature points are plotted, and the plotted points are complemented by smooth lines. As shown in this graph, the sparser the temporal distribution of the feature points, the larger the amount of movement, and the larger the change in the density of the temporal distribution of the feature points, the steeper the slope of the curve. This graph is also a graph showing time-series information of the amount of movement. The time-series information of the amount of movement is information indicating the time transition of the amount of movement.

The video buffer unit 230 receives a plurality of videos from the video acquisition means 100, temporarily holds the received plurality of videos, and sends out the video held in the video synchronization unit 260.

The time correction unit 240 obtains the time correction amount based on at least one movement amount obtained by the movement amount calculation unit 220 in each of the plurality of images. The time correction amount is used for synchronizing a plurality of images, and typically indicates an amount for correcting a time stamp indicating a shooting time in the image to be corrected. Although the time correction unit 240 does not correct the time stamp of the reference image, it may be considered that the time correction unit 240 has obtained 0 as the time correction amount of the reference image.
When at least one feature point is a plurality of feature points, the time correction unit 240 weights each evaluation value of the time shift amount calculated from the plurality of movement amounts corresponding to each of the plurality of feature points, and weights the weights. The time correction amount may be obtained based on the above. The evaluation value of the time shift amount will be described later. The time correction unit 240 may weight each of the plurality of movement amounts according to the properties of each of the plurality of feature points. The time correction unit 240 may automatically set the weight according to the extraction rate, the likelihood, or the average change amount of each feature point, or may use the manually set weight. Here, the extraction rate, the likelihood, and the average amount of change will be described. First, the extraction rate is the ratio of the number of frames that could extract the feature points to the total number of frames of at least two videos input to the feature point extraction unit 210. The higher the extraction rate, the more the feature points are extracted in many scenes of at least two images. Next, the likelihood is the information obtained incidentally when the feature points are extracted from the video using CNN, and means the probability that the extraction result of the feature points is correct. Finally, the average amount of change is the average value of the amount of movement of the feature point. The larger the average amount of change, the more actively the feature points are. Since the time correction unit 240 calculates the correction amount by adjusting the timing at which the movement amount increases or decreases, if the average change amount is large, that is, the movement amount is large on average, a more accurate correction amount is calculated. Note that you can. Summarizing the above, the higher the extraction rate, the likelihood, and the average change amount, the higher the possibility that an accurate correction amount can be calculated. Therefore, it is considered appropriate to give a relatively large weight to the feature points where at least one of the extraction rate, the likelihood, and the average change amount is high. When the dimension of the movement amount is 2 or more, that is, when the number of feature points is 2 or more, the time correction unit 240 may calculate the time correction amount by weighting each dimension.

The correction amount storage unit 250 holds the time correction amount obtained by the time correction unit 240.

The video synchronization unit 260 receives a plurality of videos from the video buffer unit 230, updates the time stamp of the correction target video according to the time correction amount held by the correction amount storage unit 250, and sets the correction target video as the reference video. Synchronize and send the synchronized video to the video output means 300. That is, the video synchronization unit 260 adjusts the timing of transmitting each of the plurality of videos according to the time correction amount.

FIG. 4 shows a configuration example of the time correction unit 240. As shown in this figure, the time correction unit 240 includes a time shift unit 241, a time shift amount evaluation unit 242, and a time shift amount storage unit 243.

The time shift unit 241 executes a time shift with respect to the time series information of the amount of movement obtained by the motion calculation unit 220-2.
When the reference image is fixed, it may be considered that the time shift unit 241 has selected one image from a plurality of images as the reference image. The time shift unit 241 obtains a time shift amount based on at least one movement amount of the reference image and at least one movement amount of each correction target image in each of the correction target images. At this time, the time shift unit 241 compares the amount of movement calculated from the same feature point, which is the amount of movement corresponding to each of the reference image and the image to be corrected. The correction target image is an image other than the reference image among a plurality of images. The time shift amount is an amount used to synchronize each of the corrected images with the reference image, and is an amount for time-shifting at least one movement amount.

The time shift amount evaluation unit 242 uses the time shift unit 241 based on the time series information of the movement amount obtained by the movement amount calculation unit 220-1 and the time series information of the movement amount of the time shift unit 241. Evaluate the validity of the time shift amount. Here, since the time-series information of the amount of movement is calculated from the two images acquired asynchronously, typically, the timing at which the value of the amount of movement increases or decreases is also asynchronous between the two images. Is. However, when the time shift unit 241 appropriately time-shifts the time-series information corresponding to the correction target video, the timing at which the value of the movement amount increases or decreases coincides between the two videos. Depending on the nature of the frame rate of the moving image, it may not be possible to time-shift the time-series information so that the timing at which the value of the amount of movement increases or decreases between the two images completely matches. Therefore, matching the timings may include a slight shift in timing.
The time shift amount evaluation unit 242 includes at least one movement amount obtained by time-shifting at least one movement amount according to the time shift amount corresponding to each correction target image, and at least one movement amount of the reference image in each correction target image. The time shift amount is evaluated based on.
[Equation 3] is a mathematical formula indicating the optimum time shift amount. The left side of [Equation 3] shows the time shift amount, and the right side shows the function of converting the maximum evaluation value into the time shift amount. score _s indicates an evaluation value indicating the appropriateness of the correction amount. As a specific example, the evaluation value is a correlation coefficient between time-series information of the amount of movement. In general, the correlation coefficient between signals having the same timing of increase / decrease in value becomes large. Therefore, when the evaluation value is a correlation coefficient, the larger the value of the evaluation value, the more appropriate the time shift amount. That is, the time shift amount evaluation unit 242 may search for the time shift amount that maximizes the correlation coefficient corresponding to the reference image and the time-shifted correction target image.

The time shift amount storage unit 243 stores the time shift amount evaluated as optimal by the time shift amount evaluation unit 242.

The video output means 300 has a function of outputting video. As a specific example, the video output means 300 is an LCD (Liquid Crystal Display).

FIG. 5 shows a hardware configuration example of the video synchronization device 200 according to the present embodiment. The video synchronization device 200 includes a computer 10. The video synchronization device 200 may be composed of a plurality of computers 10.

As shown in this figure, the computer 10 is a computer including hardware such as a processor 11, a memory 12, an auxiliary storage device 13, an input / output IF (Interface) 14, and a communication device 15. These hardware are connected to each other via a signal line 19.

The processor 11 is an IC (Integrated Circuit) that performs arithmetic processing, and controls the hardware included in the computer. As a specific example, the processor 11 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit).
The video synchronization device 200 may include a plurality of processors that replace the processor 11. The plurality of processors share the role of the processor 11.

The memory 12 is typically a volatile storage device. The memory 12 is also referred to as a main storage device or a main memory. As a specific example, the memory 12 is a RAM (Random Access Memory). The data stored in the memory 12 is stored in the auxiliary storage device 13 as needed.

The auxiliary storage device 13 is typically a non-volatile storage device. As a specific example, the auxiliary storage device 13 is a ROM (Read Only Memory), an HDD (Hard Disk Drive), or a flash memory. The data stored in the auxiliary storage device 13 is loaded into the memory 12 as needed.
The memory 12 and the auxiliary storage device 13 may be integrally configured.

The input / output IF 14 is a port to which an input device and an output device are connected. The input / output IF 14 is, as a specific example, a USB (Universal Serial Bus) terminal. The input device is, as a specific example, a keyboard and a mouse. The output device is, as a specific example, a display.

The communication device 15 is a receiver and a transmitter. As a specific example, the communication device 15 is a communication chip or a NIC (Network Interface Card).

Each part of the video synchronization device 200 may appropriately use the communication device 15 when communicating with other devices and the like.
When each part of the video synchronization device 200 receives data, each part of the video synchronization device 200 may receive data via the input / output IF 14 or may receive data via the communication device 15.

The auxiliary storage device 13 stores the video synchronization program. The video synchronization program is a program that allows a computer to realize the functions of each part of the video synchronization device 200. The video synchronization program may consist of a plurality of files. The video synchronization program is loaded into the memory 12 and executed by the processor 11. The functions of each part of the video synchronization device 200 are realized by software.

The data used when executing the video synchronization program, the data obtained by executing the video synchronization program, and the like are appropriately stored in the storage device. Each part of the video synchronization device 200 uses a storage device as appropriate. As a specific example, the storage device includes at least one of a memory 12, an auxiliary storage device 13, a register in the processor 11, and a cache memory in the processor 11. In addition, data and information may have the same meaning. The storage device may be independent of the computer 10. Each of the video buffer unit 230, the time shift amount storage unit 243, and the correction amount storage unit 250 is realized by the storage device.
Each of the function of the memory 12 and the function of the auxiliary storage device 13 may be realized by another storage device.

Each of the video acquisition means 100 and the video output means 300 may be a computer 10.

The video synchronization program may be recorded on a computer-readable non-volatile recording medium. The non-volatile recording medium is, for example, an optical disk or a flash memory. The video synchronization program may be provided as a program product.

*** Explanation of operation ***
The operation procedure of the video synchronization device 200 corresponds to the video synchronization method. Further, the program that realizes the operation of the video synchronization device 200 corresponds to the video synchronization program.

FIG. 6 is a flowchart showing an example of the operation of the video synchronization device 200. An operation example of the video synchronization apparatus 200 will be described with reference to this figure.

(Processing S01: Video acquisition processing)
The video buffer unit 230 acquires video from the video acquisition means 100.

(Process S02: Feature point extraction process)
The feature point extraction unit 210 acquires a plurality of images and extracts feature points in each of the plurality of images.

(Process S03: Motion calculation process)
The motion amount calculation unit 220 obtains the motion amount based on each of the feature points extracted in the process S02.

(Process S04: Time correction amount calculation process)
The time correction unit 240 obtains the time correction amount based on the movement amount obtained in the process S03.

(Process S05: Video synchronization processing)
The video synchronization unit 260 acquires a plurality of videos from the video buffer unit 230, synchronizes the acquired plurality of videos, and outputs the synchronized video to the video output means 300.

FIG. 7 is a flowchart showing an example of the operation of the time correction unit 240. The time correction unit 240 searches for the optimum time shift amount in the process shown in this flowchart. An operation example of the time correction unit 240 will be described with reference to this figure.

(Process S11: Search process)
This process means that the process S12, the process S13, the process S14, and the process S15 included in the search process are repeatedly executed. The time correction unit 240 repeatedly executes these processes while changing the time shift amount little by little. As a specific example, when the time length of the reference image is X seconds, the time length of the image to be corrected is Y seconds, and the time shift amount is S, the search range searched by the time correction unit 240 is −Y. <S <X, and the end condition of the search process is when S = X. The time correction unit 240 may automatically set the search range, or the user may manually set the search range when the optimum time shift amount is roughly estimated. The user is a user of the video synchronization system 90.

(Process S12: Time shift process)
The time shift unit 241 adds a time shift only to the time-series information of the amount of movement corresponding to the image to be corrected.

(Process S13: Time shift amount evaluation process)
The time shift amount evaluation unit 242 evaluates the validity of the time shift amount from the correlation coefficient between the time-series information of the movement amount of the reference image and the time-series information of the movement amount of the corrected image to which the time shift is added. .. [Equation 4] shows a specific example of the evaluation value used in [Equation 3]. [Equation 5], [Equation 6], [Equation 7], and [Equation 8] each specifically indicate a part of [Equation 4].
Cm in [Equation 4] indicates a weighting coefficient for the m-th feature point. Relative to feature points that are considered to be relatively useful in synchronizing multiple videos, such as the extraction rate, likelihood, or weight according to the average change in each feature point, or feature points that are extracted in many frames. By giving a large weight to the time shift amount evaluation unit 242, it is highly possible that an appropriate synchronization result can be obtained. When no prior information or knowledge about the feature points is given, the time shift amount evaluation unit 242 may uniformly set the value of the weighting coefficient to the same value.

(Process S14: Branch processing)
The time shift amount evaluation unit 242 compares the evaluation value obtained in the process S13 with the optimum evaluation value so far.
When the evaluation value is larger than the optimum evaluation value so far, the time correction unit 240 proceeds to the process S15. In other cases, the time correction unit 240 skips the process S15.

(Process S15: Substitution process)
The time shift amount evaluation unit 242 updates the optimum evaluation value with the evaluation value obtained in the process S13, and updates the optimum time shift amount with the amount of the time shift used in the process S12. The optimum evaluation value is an evaluation value corresponding to the optimum time shift amount. The optimum evaluation value does not have to be the ideal optimum value, and may be the best value required for practical use. The time shift amount storage unit 243 stores the optimum time shift amount and the optimum evaluation value. The optimum evaluation value and the optimum time shift amount may each refer to a value obtained in the middle of the iterative process.

The time shift amount evaluation unit 242 evaluates all the time shift amounts calculated in the search range. Therefore, the value indicated by the optimum time shift amount when the iterative process shown in the process S11 is completed is the time correction amount.

The time correction unit 240 does not need to correct the time of the video every time the video acquisition means 100 receives the video. As a specific example, the time correction unit 240 is executed only immediately after the start of use of the video synchronization system 90, or about once a day. Therefore, the correction amount storage unit 250 holds the time correction amount obtained by the time correction unit 240, and the video synchronization unit 260 is held by the correction amount storage unit 250 until the time correction process is newly executed. You may continue to use the time correction amount.

*** Explanation of the effect of Embodiment 1 ***
As described above, according to the present embodiment, it is possible to synchronize a plurality of images taken from a plurality of different viewpoints without using a state recognizer. Therefore, the range of images that can be synchronized by the video synchronization device 200 according to the present disclosure is wider than the range of images that can be synchronized by the device disclosed in Patent Document 1.
Further, according to the present disclosure, a plurality of videos can be synchronized without using an expensive device such as a synchronization server or a camera dedicated to synchronization.

*** Other configurations ***
<Modification 1>
FIG. 8 shows a hardware configuration example of the video synchronization device 200 according to this modification.
As shown in this figure, the video synchronization device 200 includes a processing circuit 18 in place of at least one of the processor 11, the memory 12, and the auxiliary storage device 13.
The processing circuit 18 is hardware that realizes at least a part of each part included in the video synchronization device 200.
The processing circuit 18 may be dedicated hardware, or may be a processor that executes a program stored in the memory 12.

When the processing circuit 18 is dedicated hardware, the processing circuit 18 is, as a specific example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (ASIC is an Application Specific Integrated Circuit), an FPGA. (Field Programmable Gate Array) or a combination thereof.
The video synchronization device 200 may include a plurality of processing circuits that replace the processing circuit 18. The plurality of processing circuits share the role of the processing circuit 18.

In the video synchronization apparatus 200, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

As a specific example, the processing circuit 18 is realized by hardware, software, firmware, or a combination thereof.
The processor 11, the memory 12, the auxiliary storage device 13, and the processing circuit 18 are collectively referred to as a "processing circuit Lee". That is, the functions of each functional component of the video synchronization apparatus 200 are realized by the processing circuit.

<Modification 2>
In the first embodiment, an example in which the reference image is fixed has been described, but the present invention is not limited to this example, and the time correction unit 240 may change the reference image depending on the situation. As a specific example, the time correction unit 240 may use an image from which the most feature points are extracted as a reference image. By doing so, the time correction unit 240 can use the image in which the subject is shown in the most frames as the reference image.
Alternatively, the time correction unit 240 may use the image of the image acquisition unit 100 at a position close to the image acquisition unit 100 of the image to be corrected as the reference image. Here, as a method for selecting the video acquisition means 100 at a close position, a method of selecting the video acquisition means 100 closest to the video acquisition means 100 of the video to be corrected, or a combination of the video acquisition means 100 is considered, and the same combination is used. A method of selecting the two closest image acquisition means 100 from a combination that does not (including the case where the reference image and the image to be corrected are exchanged) is mentioned. In this case, the time correction unit 240 calculates the time shift amount between all the correction target images and the reference image, and then adjusts the time shift amount in order to synchronize the times of all the images. Specifically, when the time shift amount is calculated with respect to the reference image, the time correction unit 240 uses the cumulative value of the value as the adjusted time shift amount. As a result, the video synchronization device 200 can compare a plurality of videos corresponding to the video acquisition means 100 having close positions to each other, so that the difference between the feature points can be reduced and the processing accuracy can be improved. be able to.

Hereinafter, this modification will be described with reference to specific examples. The video synchronization system 90 includes video acquisition means 100-1, video acquisition means 100-2, and video acquisition means 100-3 as video acquisition means 100. Further, the video A is a video acquired from the video acquisition means 100-1, the video B is a video acquired from the video acquisition means 100-2, and the video C is a video acquired from the video acquisition means 100-3.
Here, consider a set a, which is a combination in which the image A is a reference image and the image B is a correction target image, and a set b, which is a combination in which the image B is a reference image and the image C is a correction target image. The combination in which the image B is the reference image and the image A is the correction target image corresponds to the same combination as the set a.
Further, it is assumed that the time shift amount in the set a is α and the time shift amount in the set b is β. Each time shift amount shows a relative difference in each combination.
At this time, it is considered to obtain the time correction amount of the image B and the image C. At this time, assuming that the time correction amount of the image B is α and the time correction amount of the image C is β, the time reference in the image B is the time indicated by the image A, and the time reference in the image C is indicated by the image B. It is the time. That is, the time reference differs between the video B and the video C. Therefore, the time correction amount of the image C is not the time shift amount in the set b, but the cumulative value (that is, α + β) of the time shift amount in the set a and the time shift amount in the set b. The reference time in the above is the time indicated by the image A. As a result, the video synchronization device 200 can appropriately correct the time between the video A, the video B, and the video C.

*** Other embodiments ***
Although the first embodiment has been described, a plurality of parts of the present embodiment may be combined and carried out. Alternatively, the present embodiment may be partially implemented. In addition, various modifications may be made to the present embodiment as necessary, and the present embodiment may be implemented in any combination as a whole or partially.
It should be noted that the embodiments described above are essentially preferred examples and are not intended to limit the present disclosure, its applications and the scope of its use. The procedure described using the flowchart or the like may be changed as appropriate.

10 computer, 11 processor, 12 memory, 13 auxiliary storage device, 14 input / output IF, 15 communication device, 18 processing circuit, 19 signal line, 90 video synchronization system, 100 video acquisition means, 200 video synchronization device, 210 feature point extraction Unit, 220 motion amount calculation unit, 230 video buffer unit, 240 time correction unit, 241 time shift unit, 242 hour shift amount evaluation unit, 243 hour shift amount storage unit, 250 correction amount storage unit, 260 video synchronization unit, 300 video Output means.

Claims (19)

A video synchronization device that synchronizes a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, a feature point extraction unit for extracting at least one feature point among the feature points, and a feature point extraction unit.
A motion amount calculation unit for obtaining at least one motion amount indicating the amount of motion of each of the at least one feature point in each of the plurality of images.
Each of the plurality of images is provided with a time correction unit for obtaining a time correction amount used for synchronizing the plurality of images based on the at least one motion amount.
When the at least one feature point is a plurality of feature points, the time correction unit weights each of the plurality of movement amounts corresponding to the plurality of feature points, and the time correction amount is based on the weights. Video synchronization device that requires. The video synchronization device according to claim 1 , wherein the time correction unit weights each of the plurality of movement amounts according to the properties of each of the plurality of feature points. The time correction unit
One video is selected as a reference video from the plurality of videos, and in each of the correction target videos that are videos other than the reference video among the plurality of videos, the at least one movement amount of the reference video and the correction target. Time shift, which is an amount used to synchronize each of the corrected images with the reference image based on the at least one motion amount of each image, and is an amount for time-shifting the at least one motion amount. The time shift part to find the amount,
In each of the correction target images, based on at least one movement amount obtained by time-shifting the at least one movement amount according to the time shift amount corresponding to each of the correction target images, and the at least one movement amount of the reference image. With the time shift amount evaluation unit that evaluates the time shift amount
The video synchronization apparatus according to claim 1 or 2. A video synchronization device that synchronizes a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, a feature point extraction unit for extracting at least one feature point among the feature points, and a feature point extraction unit.
A motion amount calculation unit for obtaining at least one motion amount indicating the amount of motion of each of the at least one feature point in each of the plurality of images.
Each of the plurality of images is provided with a time correction unit for obtaining a time correction amount used for synchronizing the plurality of images based on the at least one motion amount.
The time correction unit
One video is selected as a reference video from the plurality of videos, and in each of the correction target videos that are videos other than the reference video among the plurality of videos, the at least one movement amount of the reference video and the correction target. Time shift, which is an amount used to synchronize each of the corrected images with the reference image based on the at least one motion amount of each image, and is an amount for time-shifting the at least one motion amount. The time shift part to find the amount,
In each of the correction target images, based on at least one movement amount obtained by time-shifting the at least one movement amount according to the time shift amount corresponding to each of the correction target images, and the at least one movement amount of the reference image. A video synchronization device including a time shift amount evaluation unit for evaluating the time shift amount. When the at least one feature point is a plurality of feature points, the time correction unit weights each of the plurality of movement amounts corresponding to the plurality of feature points, and the time correction amount is based on the weights. The video synchronization apparatus according to claim 4. A video synchronization device that synchronizes a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, a feature point extraction unit for extracting at least one feature point among the feature points, and a feature point extraction unit.
A motion amount calculation unit for obtaining at least one motion amount indicating the amount of motion of each of the at least one feature point in each of the plurality of images.
A time correction unit for obtaining a time correction amount used for synchronizing the plurality of images based on the time-series information of the at least one movement amount in each of the plurality of images.
Equipped with
When the at least one feature point is a plurality of feature points, the time correction unit weights each of the plurality of movement amounts corresponding to the plurality of feature points, and the time correction amount is based on the weights. Seeking,
The time correction unit
One video is selected as a reference video from the plurality of videos, and in each of the correction target videos that are videos other than the reference video among the plurality of videos, the at least one movement amount of the reference video and the correction target. Time shift, which is an amount used to synchronize each of the corrected images with the reference image based on the at least one motion amount of each image, and is an amount for time-shifting the at least one motion amount. The time shift part to find the amount,
In each of the correction target images, based on at least one movement amount obtained by time-shifting the at least one movement amount according to the time shift amount corresponding to each of the correction target images, and the at least one movement amount of the reference image. the time Film image synchronizer that a time shift amount evaluation unit for evaluating the shift amount. The video synchronization device according to claim 6 , wherein the time correction unit weights each of the plurality of movement amounts according to the properties of each of the plurality of feature points. A video synchronization device that synchronizes a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, a feature point extraction unit for extracting at least one feature point among the feature points, and a feature point extraction unit.
A motion amount calculation unit for obtaining at least one motion amount indicating the amount of motion of each of the at least one feature point in each of the plurality of images.
A time correction unit for obtaining a time correction amount used for synchronizing the plurality of images based on the time-series information of the at least one movement amount in each of the plurality of images.
Equipped with
When the at least one feature point is a plurality of feature points, the time correction unit weights each of the plurality of movement amounts corresponding to the plurality of feature points, and the time correction amount is based on the weights. look that movies image synchronization device. A video synchronization device that synchronizes a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, a feature point extraction unit for extracting at least one feature point among the feature points, and a feature point extraction unit.
A motion amount calculation unit for obtaining at least one motion amount indicating the amount of motion of each of the at least one feature point in each of the plurality of images.
A time correction unit for obtaining a time correction amount used for synchronizing the plurality of images based on the time-series information of the at least one movement amount in each of the plurality of images.
Equipped with
The time correction unit
One video is selected as a reference video from the plurality of videos, and in each of the correction target videos that are videos other than the reference video among the plurality of videos, the at least one movement amount of the reference video and the correction target. Time shift, which is an amount used to synchronize each of the corrected images with the reference image based on the at least one motion amount of each image, and is an amount for time-shifting the at least one motion amount. The time shift part to find the amount,
In each of the correction target images, based on at least one movement amount obtained by time-shifting the at least one movement amount according to the time shift amount corresponding to each of the correction target images, and the at least one movement amount of the reference image. the time Film image synchronizer that a time shift amount evaluation unit for evaluating the shift amount. The video synchronization according to any one of claims 1 to 9 , wherein the movement amount calculation unit obtains an amount in which the coordinate value of each of the at least one feature point changes in a unit time as each of the at least one movement amount. Device. A video buffer unit that receives the plurality of videos and temporarily holds the received multiple videos.
The video synchronization according to any one of claims 1 to 10 , further comprising a video synchronization unit that receives the plurality of images from the video buffer unit and adjusts the timing of transmitting each of the plurality of images according to the time correction amount. Device. A video synchronization method for synchronizing a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
The feature point extraction unit extracts at least one feature point among the feature points in each of the plurality of images.
The motion amount calculation unit obtains at least one motion amount indicating the amount of movement of each of the at least one feature point in each of the plurality of images.
A video synchronization method in which a time correction unit obtains a time correction amount used to synchronize the plurality of images based on time-series information of at least one movement amount in each of the plurality of images.
When the at least one feature point is a plurality of feature points, the time correction unit weights each of the plurality of movement amounts corresponding to the plurality of feature points, and the time correction amount is based on the weights. Video synchronization method to obtain . A video synchronization method for synchronizing a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
The feature point extraction unit extracts at least one feature point among the feature points in each of the plurality of images.
The motion amount calculation unit obtains at least one motion amount indicating the amount of movement of each of the at least one feature point in each of the plurality of images.
A video synchronization method in which a time correction unit obtains a time correction amount used to synchronize the plurality of images based on time-series information of at least one movement amount in each of the plurality of images.
The time correction unit
One video is selected as a reference video from the plurality of videos, and in each of the correction target videos that are videos other than the reference video among the plurality of videos, the movement amount of the reference video and the correction target Time shift, which is an amount used to synchronize each of the corrected images with the reference image based on the at least one motion amount of each image, and is an amount for time-shifting the at least one motion amount. Ask for the amount,
In each of the correction target images, based on at least one movement amount obtained by time-shifting the at least one movement amount according to the time shift amount corresponding to each of the correction target images, and the at least one movement amount of the reference image. A video synchronization method for evaluating the time shift amount . A video synchronization method for synchronizing a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
The feature point extraction unit extracts at least one feature point among the feature points in each of the plurality of images.
The motion amount calculation unit obtains at least one motion amount indicating the amount of movement of each of the at least one feature point in each of the plurality of images.
The time correction unit obtains a time correction amount used to synchronize the plurality of images based on the at least one movement amount in each of the plurality of images.
When the at least one feature point is a plurality of feature points, the time correction unit weights each of the plurality of movement amounts corresponding to the plurality of feature points, and the time correction amount is based on the weights. Video synchronization method to obtain. A video synchronization method for synchronizing a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
The feature point extraction unit extracts at least one feature point among the feature points in each of the plurality of images.
The motion amount calculation unit obtains at least one motion amount indicating the amount of movement of each of the at least one feature point in each of the plurality of images.
The time correction unit obtains a time correction amount used to synchronize the plurality of images based on the at least one movement amount in each of the plurality of images.
The time shift unit of the time correction unit selects one video as a reference video from the plurality of videos, and in each of the correction target videos which are videos other than the reference video among the plurality of videos, the reference video is described. An amount used to synchronize each of the corrected images with the reference image based on at least one motion amount and the at least one motion amount of each of the corrected images, and the at least one motion. Find the time shift amount, which is the amount that shifts the amount over time.
The time shift amount evaluation unit of the time correction unit has at least one movement amount obtained by time-shifting the at least one movement amount according to the time shift amount corresponding to each of the correction target images in each of the correction target images, and the reference. A video synchronization method for evaluating the time shift amount based on the at least one movement amount of the video. A computer that stores a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, at least one feature point among the feature points is extracted.
In each of the plurality of images, at least one amount of movement indicating the amount of movement of each of the at least one feature point is obtained.
A video synchronization program for obtaining a time correction amount used for synchronizing the plurality of images based on the time-series information of the at least one motion amount in each of the plurality of images.
When the at least one feature point is a plurality of feature points, a video synchronization program that weights each of the plurality of movement amounts corresponding to each of the plurality of feature points and causes the time correction amount to be obtained based on the weight. .. A computer that stores a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, at least one feature point among the feature points is extracted.
In each of the plurality of images, at least one amount of movement indicating the amount of movement of each of the at least one feature point is obtained.
A video synchronization program for obtaining a time correction amount used for synchronizing the plurality of images based on the time-series information of the at least one motion amount in each of the plurality of images.
One video is selected as a reference video from the plurality of videos, and in each of the correction target videos that are videos other than the reference video among the plurality of videos, the movement amount of the reference video and the correction target Time shift, which is an amount used to synchronize each of the corrected images with the reference image based on the at least one motion amount of each image, and is an amount for time-shifting the at least one motion amount. Ask for the amount,
In each of the correction target images, based on at least one movement amount obtained by time-shifting the at least one movement amount according to the time shift amount corresponding to each of the correction target images, and the at least one movement amount of the reference image. A video synchronization program that evaluates the time shift amount . A computer that stores a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, at least one feature point among the feature points is extracted.
In each of the plurality of images, at least one amount of movement indicating the amount of movement of each of the at least one feature point is obtained.
A video synchronization program for obtaining a time correction amount used for synchronizing the plurality of images based on at least one motion amount in each of the plurality of images.
When the at least one feature point is a plurality of feature points, a video synchronization program that weights each of the plurality of movement amounts corresponding to each of the plurality of feature points and causes the time correction amount to be obtained based on the weight. .. A computer that stores a plurality of images of the subject taken from a plurality of different viewpoints, including a feature point indicating a position where the coordinates change with the movement of the subject.
In each of the plurality of images, at least one feature point among the feature points is extracted.
In each of the plurality of images, at least one amount of movement indicating the amount of movement of each of the at least one feature point is obtained.
A video synchronization program for obtaining a time correction amount used for synchronizing the plurality of images based on at least one motion amount in each of the plurality of images.
One video is selected as a reference video from the plurality of videos, and in each of the correction target videos that are videos other than the reference video among the plurality of videos, the movement amount of the reference video and the correction target Time shift, which is an amount used to synchronize each of the corrected images with the reference image based on the at least one motion amount of each image, and is an amount for time-shifting the at least one motion amount. Ask for the amount,
In each of the correction target images, based on at least one movement amount obtained by time-shifting the at least one movement amount according to the time shift amount corresponding to each of the correction target images, and the at least one movement amount of the reference image. A video synchronization program that evaluates the time shift amount.

Images