JP2016070891A - Video data processor and video data processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video data processor capable of robustly estimating the position attitudes of cameras or mutual positional relationships between the cameras from video data and sensor data, and visualizing them.SOLUTION: A video data processor includes: data input means for inputting a plurality of sets of video data of an imaged video and sensor data from which the position attitude of imaging means that has imaged the video can be estimated; position attitude estimation means for estimating the position attitude of each of a plurality of imaging means from the plurality of sensor data; position attitude diagram creation means for creating and outputting a position attitude diagram indicating the position attitude of each of the plurality of imaging means from the estimated position attitude; display means for displaying the position attitude diagram and the video data on a screen; and video selection means for selecting the video data of the imaging means selected from the position attitude diagram, and outputting the video data to the display means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a video data processing apparatus and a video data processing program.

  With the spread of sensor-equipped camera devices such as smartphones, videos of events taken by audiences such as live performances and street performances are being shared on the web. And anyone can freely view a huge number of videos taken from various viewpoints of an event. Multi-viewpoint viewing services that allow users to view videos together for each event for the purpose of more comfortable viewing of these enormous numbers of videos have also appeared. Many of these multi-viewpoint viewing services use a method in which a plurality of captured video images are displayed as thumbnail images (or video images), and a user selects a video from an arbitrary viewpoint from among them and listens to it.

  However, in order for the user to select a video of a desired viewpoint from the thumbnail group in this manner, it is necessary for the user himself to estimate from which viewpoint each thumbnail image group is shot. Therefore, for example, when there are tens or hundreds of target video groups, it is difficult to quickly select a video of a desired viewpoint. In order to deal with such problems, "visualization of the positional relationship / position and orientation of the viewpoint" such as which image is taken from which position and from what position on the system side, and what positional relationship each camera is in Attempts to do so.

  This is considered to be an effective solution because it saves the user from estimating the viewpoint from the video. The positional relationship mentioned here includes a network topology in which a so-called camera is regarded as a node, such as which cameras are adjacent to each other. The position means the position of the origin of the coordinate system of the camera in the world coordinate system, and is represented by a translation vector. The posture means the direction of each axis of the coordinate system of the camera and is represented by a rotation matrix. The position and orientation refers to these translation vectors and rotation matrices.

  On the other hand, as in Non-Patent Document 1, a method for estimating the position and orientation of each camera device from a video has been proposed.

Photo Tourism: Exploring Photo Collections in 3DNoah Sanvely, Steven M. Seitz, Richard Szeliski SIGGRAPH Conference Proceedings, 2006, 1-59593-364-6, pp: 835-846, ACM Press

  However, these methods have a limitation that the shooting areas of the camera devices must overlap. It is difficult for arbitrarily shot images to have overlapping areas in all frames. In addition, the image may be deteriorated such as blur (blurring of subject) or encoding noise. When these situations occur, it is often impossible to take the feature point correspondence necessary for position and orientation estimation. That is, there is a problem in that a video that is arbitrarily shot has a large number of frames to which this position and orientation estimation method cannot be applied.

  The present invention has been made in view of such circumstances, and provides a video data processing apparatus and a video data processing program capable of robustly estimating the position and orientation of a camera or the mutual positional relationship from video data and sensor data. The purpose is to provide.

  The present invention includes a data input unit that inputs a plurality of sets of video data of a captured video and sensor data that can estimate the position and orientation of the imaging unit that captured the video, and a plurality of the imagings from a plurality of the sensor data. Position and orientation estimation means for estimating the position and orientation of each means, position and orientation diagram creation means for creating and outputting a position and orientation diagram indicating the position and orientation of each of the plurality of imaging means from the estimated position and orientation, and the position and orientation And a display means for displaying the video data on a screen, and a video selection means for selecting the video data of the imaging means selected from the position and orientation diagram and outputting the selected video data to the display means. It is characterized by that.

  The present invention further includes a relative position and orientation estimation unit that estimates a relative position and orientation of the imaging unit from the video data, and the position and orientation estimation unit estimates the imaging unit based on the relative position and orientation and the sensor data. When a mismatch occurs between the position and orientation of the image pickup unit, the position and orientation of the imaging unit estimated based on the sensor data is corrected.

  The present invention further includes a data processing unit that performs processing for synchronizing the plurality of video data and the plurality of sensor data input by the data input unit.

  The present invention is characterized in that the data processing unit performs noise removal of the sensor data before performing the synchronizing process.

  The present invention is characterized in that the data processing unit interpolates a time value at which the sensor data cannot be acquired before performing the synchronization process.

  The present invention is a video data processing program for causing a computer to function as the video data processing apparatus.

  According to the present invention, it is possible to robustly estimate the position and orientation of each camera or the mutual positional relationship using video data obtained by the camera and sensor data corresponding thereto.

1 is a block diagram showing a configuration of a video data processing apparatus 1 according to an embodiment of the present invention. It is a flowchart which shows operation | movement of the video data processing apparatus shown in FIG. It is a conceptual diagram of the method of estimating the position and orientation of a data acquisition device using video data and sensor data. It is a flowchart which shows the processing operation which estimates the position and orientation between data acquisition apparatuses using video data and sensor data. It is a block diagram which shows the detailed structure of the display part 15 shown in FIG. It is a flowchart which shows operation | movement of the display part 15 shown in FIG. It is a figure which shows an example of the window which shows the image | video which displays on a display apparatus, and the window which shows the positional relationship (bird's-eye view) of the data acquisition apparatus Di.

  Hereinafter, a video data processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a video data processing apparatus 1 according to the embodiment. As shown in FIG. 1, the video data processing apparatus 1 includes a data input unit 11, a data storage unit 12, a data processing unit 13, a position / orientation estimation unit 14, and a display unit 15.

  The data input unit 11 inputs video data to be processed and sensor data when the video is captured. The data storage unit 12 stores the input video data and sensor data. The data processing unit 13 applies preprocessing such as noise removal, interpolation, and synchronization to the video data and sensor data. The position / orientation estimation unit 14 uses the image data and sensor data to estimate the position and orientation of the data acquisition apparatus that has acquired the image data and sensor data. Here, the data acquisition device is a device that includes an image sensor that acquires video data, and various sensors such as a gyro sensor and an acceleration sensor in the same housing. Specifically, a smart phone, a tablet terminal, etc. are mentioned as an example. The display unit 15 displays a schematic diagram and a video showing the estimated position and orientation of the data acquisition device.

  Next, the processing operation of the video data processing apparatus 1 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the video data processing apparatus shown in FIG. In this embodiment, video data and sensor data are the same scene recorded by a plurality of data acquisition devices, and video refers to data having an image sequence and sound acquired by a camera and a microphone, and sensor data is Data that can estimate the position and orientation of a camera acquired by a sensor other than the camera such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor. In this embodiment, it is assumed that each image and each sensor data have a one-to-one correspondence, and both corresponding ones are recorded by the same data acquisition device, and their positions and orientations are completely the same. And

  First, the data input unit 11 inputs video data and sensor data acquired by a plurality of data acquisition devices, and stores them in the data storage unit 12 (step S1). Here, all the video data and sensor data of the all-time all-data acquisition device are input and stored, but may be input, stored, and processed by dividing every several frames. Further, it may be divided for each of several data acquisition devices.

  When the storage of the video data and the sensor data is finished, the data processing unit 13 performs noise removal processing and interpolation processing in the time direction on the sensor data for each data acquisition device (step S2). The obtained sensor data is obtained by discretely recording a time stamp indicating the time and a sensor value (sensor data) at the time. In general, a value recorded by a sensor often includes noise. Therefore, a value with less noise is obtained by applying a noise removal filter to these values. The noise removal filter used here may be performed by any process. As a general method, smoothing may be performed by applying a low-pass filter to remove fluctuations in values such as spikes, or by applying a moving average filter. If it is acceptable to allow noise in the sensor data, the noise removal process may not be performed.

  Further, the rate at which these sensor data are recorded does not match the frame rate of video and audio. Therefore, the value of the sensor data at the time of each frame of the sensor data video is generated from the recorded value of the sensor data by interpolation processing (step S3). The interpolation process used here may be performed by any method. Common methods include linear interpolation and spline interpolation. If the frame rate of the sensor data matches the frame rate of video and audio, or if the frame rate does not match but sensor data corresponding to each frame time of the video can be recorded, or the time does not match, but there is a gap Can be used, sensor data at a close time may be used as it is without performing the interpolation process.

Next, the data processing unit synchronizes time between the data acquisition devices that have acquired the video data and the sensor data (step S4). When recording is performed in advance in synchronization, it may not be performed when synchronization processing has already been performed. The synchronization process used here may be performed by any method. As a general method, there is a method in which audio and video are recorded in synchronization, and a method of performing synchronization using audio data of video (for example, see Reference 1).
Reference 1: Markerless Motion Capture with Unsynchronized Moving Cameras Nils Hasler1, Bodo Rosenhahn1, Thorsten Thormahlen ¨, Michael Wand, Juergen Gall1, Hans-Peter Seidel Computer Vision and Pattern Recognition, 2009. CVPR2009. IEEE Conference on, pp: 224-231 , ISSN: 1063-6919

  There is also a method of performing synchronization by image processing when a strobe is cooked at the time of recording. Alternatively, the synchronization may be performed manually by looking at the image data of the video.

  Next, the position / orientation estimation unit 14 obtains the position / orientation in the world coordinate system of each data acquisition device at all times from the video data and the sensor data (step S5). The position and orientation in the world coordinate system here is a position and orientation when the system composed of all data acquisition devices is the world coordinate system, and does not necessarily correspond to a physical quantity. Any method may be used for obtaining the position and orientation used here.

  As a general method, when each video has an overlapping area, a relative position and orientation between a plurality of cameras is estimated by using a method of Structure from Motion (see, for example, Non-Patent Document 1). be able to. For each camera, the relative position and orientation with one or more other cameras can be estimated, and for all cameras, if the relative positions with other cameras are found indirectly, one of the cameras can be used as a reference, It is possible to determine the position and orientation of all cameras in the world coordinate system centered on the reference camera.

  However, an arbitrarily shot video does not necessarily have an overlapping area at all times. Therefore, when the video does not have an overlapping area, the position and orientation are obtained using sensor data. With respect to a certain data acquisition device, it is possible to obtain the displacement of the posture from a certain time using a gyro sensor. If a geomagnetic sensor or an acceleration sensor is used, the posture in the East-North-Up coordinate system can always be obtained, and this may be used.

  It is possible to obtain the position and orientation of the data acquisition device at all times by estimating the transition of the position and orientation using these frames that cannot be estimated in the video. However, since these sensor data are likely to generate a cumulative error, estimation accuracy may be significantly reduced if the position and orientation is estimated using only the sensor data for a long time. Therefore, it is desirable to periodically refresh the accumulated error of sensor data. For example, when the position and orientation can be estimated from the video in a certain frame, it is possible to refresh by matching the position and orientation obtained using the sensor data with the position and orientation. As for the position of the data acquisition device, a GPS sensor may be used if an error of several tens of meters is allowed.

  Next, the display unit 15 presents the obtained position and orientation of each data acquisition device as a schematic diagram (position and orientation diagram) (step S6). Further, the display unit 15 presents an image of the selected data acquisition device (step S7). These presentations are the output of the video data processing apparatus 1.

  Next, a method for estimating the position and orientation of the data acquisition device (processing in step S5) will be described with reference to FIGS. FIG. 3 is a conceptual diagram of a method for estimating the position and orientation of the data acquisition device using video data and sensor data. D0, D1, D2, and D3 represent each data acquisition device (in the case of four data acquisition devices), and D0 represents a reference data acquisition device. T represents time. Di (i = 0, 1,...) And a dotted line in the t direction indicate that the relative position and orientation between Di at that time have not been estimated, and a solid line indicates that the estimation has been completed. The purpose of the position / orientation estimation unit is to make all dotted lines existing between Di using a video data and sensor data into a solid line.

  FIG. 4 is a flowchart showing a processing operation for estimating the position and orientation between data acquisition devices using video data and sensor data. First, a data acquisition device to be a reference D0 is determined (step S11). D0 may be determined by any method. For example, there is a method of manually selecting or automatically selecting a data acquisition device having the smallest displacement in the time direction based on sensor data.

  Next, the relative position and orientation of each Di (i = 1, 2,...) With respect to the reference D0 is obtained using the video (step S12). As a method for obtaining the relative position and orientation from the video, Structure from Motion may be used. However, Structure from Motion uses feature point correspondence between videos, so if the video does not have overlapping regions, or if correct feature point correspondence cannot be obtained due to blur, coding noise, etc., the position and orientation There are cases where the estimation accuracy is significantly reduced or estimation is impossible. This corresponds to the fact that a solid line between D0 and Di cannot be obtained at a certain time in FIG.

  By performing the process of step S12 for all times (step S13), the relative position and orientation can be obtained at some times between D0 and Di. That is, in FIG. 3, the dotted line between D0 and Di is partly a solid line. Further, even when the relative position and orientation with respect to D0 are not directly obtained as in D2 in FIG. 3, if the relative position and orientation with D1 with which the relative position and orientation is directly obtained with respect to D0 is obtained, D1 is determined. It is possible to obtain the relative position and orientation with respect to D0 by going through. In this embodiment, a data acquisition device in which the relative position and orientation with respect to D0 cannot be obtained through any data acquisition device is excluded.

  Next, a change in position and orientation in the time direction is obtained for each Di (step S14). This can be estimated using video data or sensor data. Any method may be used as the estimation method. When using an image, Structure from Motion may be used in the time direction. When using sensor data, a gyro sensor or the like may be used. The gyro sensor records the displacement with respect to each axis per unit time in the coordinate system of the sensor. It is possible to obtain a change in the relative posture with respect to the time, which is to add and accumulate the actual time. In addition, the acceleration sensor determines the acceleration for each axis per unit time. It is possible to obtain a change in relative position with respect to time, which is to add and accumulate the actual time. By performing this process for all times and all Di (step S15), the displacement of the position and orientation in the time direction of Di can be obtained. That is, in FIG. 3, it means that the dotted line in the t direction is a solid line.

  Note that the position and orientation obtained by using these sensor data are the position and orientation with respect to the coordinate system of the sensor, and do not necessarily match the coordinate system of the camera. However, if the system is such that the camera and sensor are mounted in the same housing, such as a smartphone, and the relative position is unchanged and the difference in relative position is negligible, the change in the position and orientation of the sensor data It may be seen as a change in.

  Next, the position and orientation mismatch is corrected (step S16). As shown in FIG. 3, when the positions and orientations of D0 and D1 are obtained from the video at times ti and tj, sensor data is used from the position and orientation obtained from the video at time tj and the position and orientation obtained from the video at time ti. The estimated values do not always match. This is largely due to the accumulated error of the sensor. Although it is not necessary to make these values coincide, it becomes possible to refresh the accumulated error of the sensor. Any method may be used as a method for matching these values. For example, matching may be performed using the following method.

The angle notation of the posture obtained from the video at time t is (xm (t), ym (t), zm (t)), and the angle notation of the relative posture in the time direction obtained from the sensor data from time t0 to time t1. Is (xs (t0, t1), ys (t0, t1), zs (t0, t1)). After the position and orientation (xm (ti), ym (ti), zm (ti)) are obtained from the video at time ti, the position and orientation (xm (tj), ym (tj)) from the video at time tj (> ti). , Zm (tj)) is obtained, (xm (tj), ym (tj), zm (tj)) and (xm (ti), ym (ti), zm (ti)) + (xs (ti) , Tj), ys (ti, tj), and zs (ti, tj)) are matched with each other for (xs (ti, t), ys (ti, t), zs (ti, t)) at each time. The following coefficients should be applied.
ax = (xm (tj) −xm (ti)) / xz (ti, tj)
ay = (ym (tj) −ym (ti)) / yz (ti, tj)
az = (zm (tj) -zm (ti)) / zz (ti, tj)

  Finally, (xs (ti, t), ys (ti, t), zs (ti, t)) multiplied by each coefficient may be changed into a rotation matrix. The same correction method can be applied to the position. This process is performed for all mismatches (step S17).

  Next, the relative position and orientation between Di with respect to D0 at each time is obtained (step S18). Now, the position and orientation of each Di is obtained at D0 and at any time, and the displacement of the position and orientation within each Di is obtained. Therefore, even if the position and orientation of D0 and Di are not obtained from the video at a certain time, it is possible to go back to the time when the position and orientation are obtained from the video and add the position and orientation displacement at each time based on the position and orientation. Good. This process is performed for all times and Di (step S19). The above is the processing operation for obtaining the position and orientation.

  Next, the processing operation in which the display unit 15 presents the obtained position and orientation diagram and video of each data acquisition device as the output of the multi-view video viewing system (processing operation in steps S6 and S7) is shown in FIGS. This will be described with reference to FIG. First, the detailed configuration of the display unit 15 shown in FIG. 1 will be described with reference to FIG. FIG. 5 is a block diagram showing a detailed configuration of the display unit 15 shown in FIG. As shown in FIG. 5, the display unit 15 includes a display unit data input unit 151, a position / orientation diagram creation unit 152, a presentation video determination unit 153, and an external input reception unit 154. The display unit data input unit 151 inputs the video acquired by the data acquisition device (output of the data processing unit 13) and the position and orientation (output of the position and orientation estimation unit 14). The position / orientation diagram creation unit 152 creates a schematic diagram representing the position / orientation of the data acquisition device. The presentation video determination unit 153 determines a video to be presented. The external input receiving unit 154 receives the user's operation content as an external input.

  Next, the operation of the display unit 15 shown in FIG. 5 will be described with reference to FIG. FIG. 6 is a flowchart showing the operation of the display unit 15 shown in FIG. First, the display unit data input unit 151 inputs the video acquired by the data acquisition device and the position and orientation of the data acquisition device Di (step S21). Subsequently, the position / orientation diagram creation unit 152 generates a position / orientation diagram (schematic diagram) indicating the position of each data acquisition device at each time from the input position / orientation data (step S22). . Any method may be used to display the position and orientation diagram. For example, as shown in FIG. 7B, a bird's eye view of the data acquisition device is created. FIG. 7 is a diagram illustrating an example of a window that presents a window (FIG. 7A) that presents an image to be displayed on the display device and a positional relationship (bird's eye view, FIG. 7B) of the data acquisition device Di. .

  Next, an image and a position / orientation diagram of the selected data acquisition device (hereinafter also referred to as viewpoint) are presented (step S23). The presenting method may be any method. For example, as shown in FIG. 7, two windows are presented, and an image and a position / orientation diagram are presented for each of them. When a viewpoint is not selected (such as an initial state), the viewpoint may be automatically given.

  Next, the external input receiving unit 154 receives a viewpoint selection (step S24). As a method for selecting a viewpoint, a method of accepting from an external input by a user operation and a method of selecting automatically are conceivable. If automatic, the viewpoint to be selected may be changed after a certain period of time. Also, by using a tracking method, the viewpoint may be automatically selected so that a certain person is always captured. Regarding external input, a desired camera may be selected and input on the position and orientation diagram using an external interface such as a keyboard or a mouse. Then, it is determined whether viewpoint selection has been input (step S25). If it has been input, the process returns to step S23 to repeat the process. On the other hand, when the viewpoint selection is not input and the end condition is input, the processing operation of the system is ended (step S26).

  In this way, it is possible to create a multi-view video viewing system that provides a more comfortable viewing by robustly estimating the position and orientation using video and sensor data acquired by a plurality of data acquisition devices.

  In the above-described embodiment, the position and orientation composed of the rotation matrix and the translation vector are estimated for each camera and used for visualization. However, the relationship between the cameras is described in another method, and the relationship is estimated. It is good as well. For example, a graph structure having each camera as a node is constructed, and the physical distance is estimated by giving a physical distance as a distance between the nodes. The display device may visualize the graph and realize viewpoint switching by designating a node position on the graph or transitioning to an adjacent node along an edge.

  As described above, a multi-viewpoint video viewing system that robustly estimates the position and orientation of each camera or the positional relationship between them by using the video of a camera device such as a smartphone and the sensor data attached to the device, and visualizes them. It aims at realizing.

  According to the present invention, it is possible to robustly estimate the position and orientation of each camera or the positional relationship between them using video data obtained by a camera and corresponding sensor data. In addition, the present invention can provide a multi-view video viewing system that visualizes these positions and orientations and uses them for viewing. These allow the user to quickly select a desired viewpoint image from a large number of images.

  You may make it implement | achieve the video data processing apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  The present invention can be applied to applications where it is indispensable to robustly estimate the position / orientation of each camera or the positional relationship between them from video data and sensor data and visualize them.

  DESCRIPTION OF SYMBOLS 1 ... Video | video data processing apparatus, 11 ... Data input part, 12 ... Data storage part, 13 ... Data processing part, 14 ... Position and orientation estimation part, 15 ... Display part, 151 ... Display unit data input unit, 152 ... Position and orientation diagram creation unit, 153 ... Presentation video determination unit, 154 ... External input reception unit

Claims (6)

Data input means for inputting a plurality of sets of video data of the captured video and sensor data capable of estimating the position and orientation of the imaging means for capturing the video;
Position and orientation estimation means for estimating the position and orientation of each of the plurality of imaging means from the plurality of sensor data; and
Position and orientation diagram creating means for creating and outputting a position and orientation diagram indicating the position and orientation of each of the plurality of imaging means from the estimated position and orientation;
Display means for displaying the position and orientation diagram and the video data on a screen;
A video data processing apparatus comprising: video selection means for selecting the video data of the imaging means selected from the position and orientation diagram and outputting the selected video data to the display means. A relative position and orientation estimating means for estimating a relative position and orientation of the imaging means from the video data;
The position and orientation estimation unit is configured to estimate the position and orientation of the imaging unit based on the sensor data when a mismatch occurs between the relative position and orientation and the position and orientation of the imaging unit estimated based on the sensor data. The video data processing device according to claim 1, wherein the video data processing device is modified.   3. The video data processing apparatus according to claim 1, further comprising a data processing unit configured to perform a process of synchronizing a plurality of the video data input by the data input unit and a plurality of the sensor data. .   The video data processing apparatus according to claim 3, wherein the data processing unit performs noise removal of the sensor data before performing the synchronization process.   5. The video data processing apparatus according to claim 3, wherein the data processing unit interpolates a time value at which the sensor data cannot be acquired before performing the synchronization process. 6.   A video data processing program for causing a computer to function as the video data processing device according to any one of claims 1 to 5.

Images