KR20210022279A - Motion sensor-based approach method and apparatus for automatic capture and editing of photos and videos - Google Patents

Abstract

Presented are motion sensor-based approach and device for automatically taking and editing a photo and video using a smartphone. The motion sensor-based approach according to one embodiment of the present invention may comprise: a synchronization step in which a first terminal and a second terminal are connected through wireless communication; a data acquisition step of recording a video with a camera of the first terminal; and a data processing step of receiving recorded motion sensor data from the second terminal possessed by a person or object to be photographed, and analyzing and detecting the motion sensor data.

Description

Motion sensor-based approach and device for automatic photographing and editing of photos and videos {MOTION SENSOR-BASED APPROACH METHOD AND APPARATUS FOR AUTOMATIC CAPTURE AND EDITING OF PHOTOS AND VIDEOS}

The following embodiments relate to a motion sensor-based approach and apparatus for automatically capturing and editing photos and videos using a smartphone.

Since photography was invented in the 19th century, photography and video shooting was not straightforward until about 40 years ago. Expensive equipment used by professionals was required, it was for an important event, and light-sensitive film had to be developed on a physical pedestal to view the final picture. Also, video editing involved using a razor blade to physically cut the film and attach a piece of film.

With the recent advent of smartphones, taking pictures and videos has become very easy, virtually inexpensive, and is now a part of everyday life. However, despite the ease of use of a smartphone, it is still difficult to take a photo of a short motion at an accurate point in time, and video editing is a manual and time-consuming task performed by an experienced user.

A typical time-consuming video editing task is modifying the partial speed of motion in the video (also known as speed ramping). For example, the part of the video where a tennis player hits the ball is slow, while the rest of the video is at normal speed. Specifically, the user must manually select the moment of hitting the ball (viewing the entire video) and use sophisticated video editing software to correct for changes in speed. Also, this process must be repeated every time the video ball is hit, and for each input video. Similarly, a typical photographing that is difficult to obtain is a jump photograph. Jumping photos are very popular, but jumping is a very short motion, so it's difficult to take a picture at the exact point in time, so it actually takes several attempts. This is due to relatively long trigger delays, especially for smartphones.

As such, with the advent of smartphones, taking pictures and videos has become very easy, and it is now a part of everyday life. However, it is still difficult to shoot short motions (e.g., jump photos) at the correct time, and video editing (e.g. some local slow motion) is a manual and time-consuming task for experienced users.

Dan Morris, T. Scott Saponas, Andrew Guillory, and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find, recognize, and count repetitive exercises. In ACM Human Factors in Computing Systems (CHI). 3225-3234.

The embodiments describe a motion sensor-based approach and apparatus for automatic photographing and editing of photos and videos using a smartphone, and more specifically, the ubiquity of a smartphone and a smartphone in order to facilitate the photographing and editing operation. It offers a technology that takes advantage of its advanced technology features (especially built-in motion sensors, high frame rate cameras and wireless connectivity).

Embodiments are motion for automatic photographing and editing of photos and videos using a smartphone that can automatically "snap" the video editing effect intended by the user to the input video using motion sensor data of the captured video. It is to provide a sensor-based approach and device.

A motion sensor-based approach method according to an embodiment includes: a synchronization step in which a first terminal and a second terminal are connected through wireless communication; A data acquisition step of recording a video with a camera of the first terminal; And a data processing step of receiving recorded motion sensor data from the second terminal possessed by a person or object to be photographed, and analyzing and detecting the motion sensor data.

In the synchronization step, the timeline of the motion sensor data may be temporally aligned with the timeline of the video by synchronizing the first terminal and the second terminal.

The data processing step includes receiving an operation time for the motion sensor data at a moment when acceleration of more than a threshold value occurs from the second terminal or a moment when the direction of the second terminal coincides with a target direction, and at the operation time. The corresponding video frame or segment can be automatically selected.

The data processing step may further include setting the threshold value through an adhoc processing technique.

The data processing step may further include providing at least one video effect of slow motion, camera selection, text superimposition, audio effect, and music to a video frame or segment corresponding to the motion time.

The data processing step may include detecting two times having the highest vertical acceleration acquired by the second terminal possessed by a person to be photographed in order to automatically calculate the time of the jump position; And recognizing the two sensed times as jumping and landing, recognizing an intermediate time between the two times as a jump time, and returning a corresponding video frame.

The data processing step may include: detecting a peak of acceleration acquired by the second terminal possessed by a person or object to be photographed in order to automatically calculate a hit time; And recognizing the detected acceleration peak as a hit time, and returning a corresponding video frame.

The second terminal may acquire accelerometer or gyroscope data.

The first terminal and the second terminal are smart phones, and can be automatically edited after taking a video by using the smart phone.

According to another embodiment, a motion sensor-based access device includes: a synchronization unit to which a first terminal and a second terminal are connected through wireless communication; A data acquisition unit for recording video with a camera of the first terminal; And a data processing unit that receives recorded motion sensor data from the second terminal possessed by a person or object to be photographed, and analyzes and senses the motion sensor data.

The synchronization unit may synchronize the first terminal and the second terminal to temporally align the timeline of the motion sensor data with the timeline of the video.

The data processing unit receives an operation time of the motion sensor data at a moment when acceleration of more than a threshold value occurs from the second terminal or a moment when the direction of the second terminal coincides with a target direction, and corresponds to the operation time. You can automatically select the video frame or segment you want.

The data processing unit may provide at least one video effect of slow motion, camera selection, text superposition, audio effect, and music to a video frame or segment corresponding to the operation time.

The data processing unit detects two times having the highest vertical acceleration acquired from the second terminal possessed by the person to be photographed in order to automatically calculate the time of the jump position, and leaps the detected two times. And recognizing as a landing, recognizing an intermediate time between the two times as a jump time, and returning a corresponding video frame.

The data processing unit detects a peak of acceleration obtained from the second terminal possessed by a person or object to be photographed in order to automatically calculate a hit time, and calculates the detected peak of the acceleration as a hit. It can recognize visually and return a corresponding video frame.

The first terminal and the second terminal are smart phones, and can be automatically edited after taking a video by using the smart phone.

According to embodiments, automatic photographing and editing of photos and videos using a smartphone that can automatically "snap" the video editing effect intended by the user to the input video using motion sensor data of the captured video. It is possible to provide a motion sensor-based approach and apparatus for

According to embodiments, by applying a motion effect (eg, some slow motion) at a precise point in time, automatic photos and videos using a smartphone that automatically operate without performing visual tracking or video analysis. A motion sensor-based approach and apparatus for shooting and editing can be provided.

1 is a diagram illustrating examples of a motion sensor-based approach according to an embodiment.
2 is a diagram for describing a situation in which a specific motion is photographed using a smartphone camera according to an exemplary embodiment.
3 is a flowchart illustrating a motion sensor-based approach method according to an embodiment.
4 is a block diagram illustrating a motion sensor-based access device according to an exemplary embodiment.
5 is a diagram for describing price detection and jump detection according to an exemplary embodiment.
6 is a diagram for explaining automatic camera selection in setting a plurality of cameras according to an exemplary embodiment.
7 is a diagram for describing a motion sensor-based approach to a plurality of motion sensors in the context of a group jump photograph according to an exemplary embodiment.
8 is a diagram illustrating an arrangement of a smartphone for data capture for a bullet time effect according to an exemplary embodiment.
9 is a diagram illustrating an example of a bullet time point according to an embodiment.
10 is a diagram illustrating an example of gyro-based browsing according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various different forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more completely describe the present invention to those of ordinary skill in the art. In the drawings, the shapes and sizes of elements may be exaggerated for clearer explanation.

To solve the shooting and editing problem, you can make two observations. First, smartphones are ubiquitous. For example, in 2016, 90% of the U.S. population between the ages of 18 and 49 have a smartphone. That said, almost everyone has a smartphone, which suggests bringing them together. Second, the smartphone is a beautiful piece of condensation technology with advanced sensors, computing resources, storage and connectivity capabilities. Based on these two observations, we will review how to use the smartphone's ubiquity and advanced technology features (especially built-in motion sensors, high frame rate cameras and wireless connectivity) to facilitate shooting and editing tasks. I can. In general, ubiquitous, connected, multi-sensor computer devices can provide a new opportunity to solve problems related to media content editing in a simple way, which is in this embodiment in the context of shooting and editing. This is the content to be presented.

As the first step for automatic shooting and editing, the present embodiment mainly focuses on movements with short, strong acceleration, including movements such as jumping, kicking, dancing, swinging and throwing. It also shows applications for rotational motion and direction, and applications such as object upright detection and viewpoint estimation. Embodiments may automatically “snap” the video editing effect intended by the user to the input video using motion sensor data of the captured video. In other words, it is possible to apply a motion effect (eg, partial slow motion) at a precise point in time. In video editing software, the input video is structured into bars using video tracks within the editor timeline. The user can manually select the start/end point of the effect or set the video effect by dragging-and-stretching the effect toolbar.

Fundamentally, in this embodiment, it is considered as adding a motion sensor data track on the editor timeline and snapping a video effect to the video based on the motion data so that the viewpoint of the video effect is automatically set according to the motion sensor data. I can.

1 is a diagram illustrating examples of a motion sensor-based approach according to an embodiment.

Referring to FIG. 1, the motion sensor-based approach method according to an embodiment uses not only the ubiquity of a smartphone, but also its latest features (especially, a built-in motion sensor, a high frame rate camera, and a wireless connection) to take pictures and Enables the shooting and editing of videos. Accordingly, a motion sensor-based approach was implemented as a smartphone app and focused on motions with short and strong acceleration.

For example, a photo of an automatic jump as shown (a), a photo of a group jump in the combined photos "triggered" by multiple smartphone motion sensors (b), a player playing table tennis The effect of automatically displaying the "BOOM" logo every time the ball is hit (c), dance beautification (d) with the dancer's motion aligned to the beat of the music, and multiple smartphone cameras. It can be applied to a variety of situations and applications, such as the bullet time effect (e, f), and automatically adding music to the video portion of a basketball player dunk (g).

2 is a diagram for describing a situation in which a specific motion is photographed using a smartphone camera according to an exemplary embodiment.

2(a) shows a representative setting for a jump photo situation in which a jumping person puts a smartphone in a pocket, and (b) is a photo automatically acquired by the motion sensor-based approach method according to an embodiment. Show. That is, the smartphone of the jumping person and the smartphone of the person taking a jump picture may communicate with each other to obtain a jumped motion.

And, (c) shows the setting of the situation in which the motion smartphone is attached to the sandbag. Accordingly, in the motion sensor-based approach according to an embodiment, the hit is slow motion, while the remaining video portion may partially retime the automatically captured video so that it becomes a normal speed. .

The task of snapping, for example, was previously only in the context of non-linear multi-video synchronization. Instead, an embodiment may review a method of applying a video effect to an operation point of view and provide a general algorithm that can be applied to various challenging and time-consuming applications such as partial slow motion, camera view selection, and frame selection.

A general algorithm can consist of three key steps. That is, it may consist of data acquisition, data processing, and synchronization. Hereinafter, a general algorithm of the motion sensor-based approach method according to the present embodiment will be described in more detail with reference to FIG. 1.

3 is a flowchart illustrating a motion sensor-based approach method according to an embodiment.

Referring to FIG. 3, in the motion sensor-based approach method according to an embodiment, a synchronization step in which a first terminal and a second terminal are connected through wireless communication (S110), data acquisition for recording video with a camera of the first terminal Step S120 and a data processing step S130 of receiving recorded motion sensor data from a second terminal possessed by a person or object to be photographed, and analyzing and detecting the motion sensor data.

Each step of the motion sensor-based approach method according to this embodiment may be described in more detail as an example of the motion sensor-based approach device according to an embodiment.

4 is a block diagram illustrating a motion sensor-based access device according to an exemplary embodiment.

Referring to FIG. 4, a motion sensor-based access device 400 according to an embodiment may include a synchronization unit 410, a data acquisition unit 420, and a data processing unit 430. For example, the motion sensor-based access device 400 according to an embodiment may be performed in a first terminal or performed through a first terminal.

In the synchronization step (S110), the synchronization unit 410 may connect the first terminal and the second terminal through wireless communication. More specifically, the synchronization unit 410 may temporally align the timeline of the motion sensor data with the timeline of the video by synchronizing the first terminal and the second terminal.

The synchronization step aims at temporally aligning the motion sensor data timeline with the video timeline, since the first terminal and the second terminal may not be synchronized. When data is synchronized, as illustrated in FIG. 2, a video frame or segment corresponding to the detected operation time may be automatically retrieved.

In the data acquisition step S120, the data acquisition unit 420 may record a video with the camera of the first terminal.

In the data acquisition step, as shown in FIG. 2, not only video is recorded with a remote smartphone camera (first terminal), but also motion sensor data is simultaneously recorded with a smartphone (second terminal) possessed by a person or object of interest. Can be recorded.

In the data processing step S130, the data processing unit 430 may receive the recorded motion sensor data from the second terminal possessed by the person or object to be photographed, and analyze and detect the motion sensor data. In the data processing step, for example, the motion sensor data can be analyzed to detect the moment when strong acceleration occurs or the moment the direction of the second terminal coincides with the target direction through a simple thresholding. I can. A sophisticated human behavior recognition method based on motion sensor data can be applied to [Non-Patent Document 1], but a simple adhoc processing technique (threshold value setting, etc.) can be provided for several reasons as follows. It is computationally efficient (important in smartphones), can be applied to a variety of use contexts, does not require training datasets, and is accurate (no window-level classification time).

More specifically, the data processing unit 430 receives the operation time for the motion sensor data at the moment when acceleration of more than a threshold value occurs from the second terminal or the moment when the direction of the second terminal coincides with the target direction, The corresponding video frame or segment can be automatically selected. In this case, the data processing unit 430 may set a threshold value through an adhoc processing technology.

Thereafter, the data processing unit 430 may provide at least one video effect of slow motion, camera selection, text superposition, audio effect, and music to a video frame or segment corresponding to the operation time.

As an example, in order to automatically calculate the time of the jump position, the data processing unit 430 detects two times having the highest vertical acceleration acquired from the second terminal possessed by the person to be photographed, and then detects the two detected times. Time is recognized as a jump and landing, and an intermediate time between two times is recognized as a jump time, and a corresponding video frame can be returned. Here, the second terminal may acquire accelerometer or gyroscope data.

As another example, in order to automatically calculate the hit time, the data processing unit 430 detects the peak of the acceleration acquired by the second terminal possessed by the person or object to be photographed, and then price the peak of the detected acceleration. (hit) Can recognize visually and return a corresponding video frame.

The first terminal and the second terminal may be a smart phone, and the video may be automatically edited after a video is captured using a smart phone. In addition, the first terminal and the second terminal are not only smartphones, but also mobile phones, portable multimedia players (PMPs), mobile Internet devices (MIDs), navigation systems, desktops, tablet computers, and notebooks. It may be a mobile terminal having various mobile communication specifications, such as, a net book and an information communication device. In the following, for convenience, the first terminal and the second terminal will be used as a smartphone or a phone.

Based on the general algorithm of the motion sensor-based approach according to this embodiment, various applications including jump photo, frame selection, automatic playback speed control of slow motion video, motion-based in video-text superposition and dance video beautification. For this, several extensions added to this algorithm are described (see Fig. 1). In addition, it is possible to extend a motion sensor-based approach with multiple people, multiple motion sensors and multiple cameras for camera point selection in multiple camera settings, bullet time effects, and group jump photos in a combined scene. have.

The motion sensor-based approach is entirely based on motion sensor data, and does not perform visual tracking and video analysis. Thus, it is fast and robust to any visual condition such as lighting, occlusion, camera orientation and position, and blur. Overall, the motion sensor-based approach takes advantage of the advanced technology features of the latest smartphones (especially motion sensors and high frame rate cameras and wireless connectivity) to solve difficult video editing tasks in a simple, automatic, fast and robust way. This is a general approach. As an example, a motion sensor-based approach is implemented as an app for an Android smartphone, but is not limited thereto. It's intuitive to use, runs in a fully automatic way, and uses a variety of video effects to deliver visually appealing results (slow motion, camera selection, text overlay, audio effects, music). The excellence of the results is successfully demonstrated using qualitative and quantitative assessments.

In order to carry out the relevant research review in the context of application and to explain the technical difficulties, we will first consider the following two use cases mentioned above. This is a partial speed-editing of a jump photo and a video of a tennis player hitting the ball (i.e. the hit scene is in slow motion while the rest of the video is at normal speed). From these two motivational examples, it is possible to present a way to generalize to various applications in the context of automatic shooting and video editing.

Automatically taking pictures of a situation or action (eg, a jump) involves automatic camera triggering. Depending on the application situation (e.g. sports, animal trap photography), the application situation may include light barriers, pressure pads, force platforms, contact mats and This can be achieved using additional hardware such as photoelectric cells. However, this additional hardware is generally expensive and difficult to transport, and users rarely use all of these devices. Users only want a way to use commonly available, ubiquitous, and portable devices.

Instead of using additional hardware, the camera can be operated to automatically take pictures by analyzing the visual content captured by the camera using real-time image processing techniques. In the context of jump photography, a probable approach could be to track the jumping person's face in a real-time video stream of a smartphone camera during the jump, activating the camera as the face begins to go down in the image. However, this simple approach has several limitations. First, photos will be taken too late due to inevitable delays, such as delays in image processing and camera operation, especially in smartphones. Second, face detection and tracking is very sensitive to lighting, motion blur and occlusion. Also, this approach assumes that the view at which the face is at the highest position in the image corresponds to the view at which the jumping person is at the highest position, but this is only valid when the jump is parallel to the camera plane in front. On the other hand, the motion sensor-based approach according to the present embodiment overcomes all these limitations and can be applied to any camera direction.

A way to address the delay is to predict when the jumping person will reach the highest position. Accordingly, it is possible to predict the time of the highest position by tracking a person's face in real time, estimating the speed of the person, and substituting it into a ballistic motion model. By subtracting the priori delay from the predicted time, it is possible to operate the camera at the correct time. However, like the above method, this approach is practically error-prone due to visual tracking and cannot solve general jump styles and camera poses. Also, you need to know the processing delay and the camera delay first, but this can vary greatly depending on several factors such as scene lighting, camera focus, computational resources, CPU usage, phone model, etc. It is difficult. Finally, since the camera works during the jump, it requires very fast connections between phones and very fast processing of motion data. In contrast, the motion sensor-based approach according to the present embodiment can overcome all of these limitations and can be generalized not only to jump photos, but also to various applications such as automatic video time resetting and camera viewpoint selection.

An alternative method is to record the jump in video or burst mode, and then use manual or automatic computer vision techniques to select the frame in the highest position by the jumping person. Manual selection is a common process in continuous mode. It's easy to use, but it doesn't provide a photo directly (requires manual interaction through scrolling), it's not scalable, and it can't solve group jump photos. More importantly, it cannot be generalized to other applications for video editing (such as the dance retime setting in Fig. 1).

Given an existing collection of photos, several automatic methods for the collection of photos have been proposed. However, the methods are designed for screening a series of sparse shots or require a dedicated training dataset. In contrast, in the motion sensor-based approach according to this embodiment, the data for jump is a very densely sampled set among video frames for a very short motion (less than 400 ms), and the motion sensor-based approach is a machine learning procedure. You don't need to create a specific dataset for it.

As an alternative to this, by implementing computer vision techniques such as optical flow, motion detection from face detection/tracking and 3D human pose estimation, video frames can be automatically selected from the captured video. However, for the same reasons as in the vision-based camera trigger described above, it is sensitive to errors in visual tracking and detection (occlusion, blurring, lighting, etc.). However, it can only solve front-to-parallel motion, it can only be applied to humans, and cannot handle moving cameras (or will require more sophisticated camera-background motion separation). Instead, the motion sensor-based approach according to this embodiment can simply overcome all these difficulties. It will also be shown that by using simple motion data processing, it is possible to handle many types of applications, from jump photos to video retiming and camera viewpoint selection.

Hereinafter, a motion sensor-based approach according to the present embodiment will be described.

To overcome the difficulties of vision-based methods, several methods have proposed using motion sensor data and are effective for some difficult computer vision and computer graphics tasks such as 3D reconstruction and video stabilization. Showed a point. In the context of camera triggering, the triggering of a previously throwable panoramic camera ball is described using an accelerometer sensor. The user throws the camera ball into the air and the camera takes a panoramic picture when it reaches the highest position. However, the motion sensor is attached directly to the camera system.

Instead, in the motion sensor-based approach according to this embodiment, the camera is generally far from the motion sensor in order to take a video or picture of an object or person of interest. For this, communication and synchronization between devices must be performed. In addition, in the motion sensor-based approach, we would like to propose a rational solution that is commonly used and uses ubiquitous smartphones. In addition, it can be shown that a motion sensor-based approach can be used for various applications in photography and video editing. In the same way as the vision-based prediction method for jump photos described previously, a single picture is taken by predicting the time of the highest point in order to operate the photo camera at an appropriate time, but motion sensor data is used. . It is more robust than the vision approach, but with the same limitations, i.e. it presumes processing delay and camera delay, and requires very fast processing and connection. It also performs integration on the acceleration data to obtain the velocity and position. However, an error occurs in the acceleration value due to the vibration that inevitably occurs in human motion, which increases exponentially during the integration process.

In contrast, the motion sensor-based approach according to the present embodiment selects a frame from the recorded jump video without performing integration, and consequently provides a robust method.

It explains the main differences from the latest technologies in existence. First, the technologies trigger the camera to take a picture, which requires knowing the camera trigger delay, which can vary depending on the lighting settings, auto focus, phone model, memory, etc. Therefore, it is very difficult to know exactly first. In contrast, the motion sensor-based approach according to the present embodiment automatically selects an optimal frame by recording a video of a jump, etc., so that all problems related to operation delay can be overcome.

In the following, a general algorithm of a motion sensor-based approach according to this embodiment for automatic photographing and editing is introduced. The main steps are data capture, data processing and device synchronization. Based on this general algorithm, it is possible to describe the extension to multiple motion sensors and multiple cameras.

5 is a diagram for describing price detection and jump detection according to an exemplary embodiment.

Referring to FIG. 5, a motion sensor-based approach for automatically calculating the time of the highest jump position is for explaining the price detection (a) and jump detection (b) for a boxing punch. It can detect two perspectives with acceleration. This corresponds to a leap and landing, and is indicated in red. Then, the intermediate time of the jump is calculated and a video frame corresponding to that time is returned. That is, it returns the frame at the highest position, as shown by the red outline.

To put an overview of the motion sensor-based approach according to an embodiment in context, consider a use case of local video retiming for a boxer's punch. The goal is to get a video with a partial video rate adjusted (local speed ramping), i.e. slow motion at punch or price, but the rest of the video at normal speed.

As discussed when reviewing related studies, taking pictures of hits (e.g., via vision analysis or motion sensors) due to inevitable, variable, and unknown data processing and delays in camera operation. It is not satisfactory to operate the camera automatically at the correct time to do so. To overcome these limitations, a motion sensor-based approach records the athlete's video and automatically selects the optimal video frame from the motion sensor data.

Specifically, the general algorithm of the motion sensor-based approach according to the present embodiment consists of the following key steps. First, in the synchronization step, the smartphones can be synchronized over a wireless connection to obtain temporally aligned video and motion sensor data. Second, in the data acquisition step, a video of a tennis player and motion sensor data can be recorded with a smartphone. Afterwards, in the offline data processing stage, i.e., right after the game, the motion sensor data is analyzed directly with a smartphone (by setting a simple threshold value on the acceleration data) to detect the time when the hit has occurred. can do. Next, the detected time can be wirelessly transmitted to the camera phone to automatically select a video frame corresponding to the time. Finally, this video frame result can be displayed on the user's smartphone. The whole process is carried out in a completely automatic way, and the resulting picture is displayed approximately 2 seconds (total processing time) after stopping video recording.

In the data acquisition section, it is possible to record motion sensor data from a smartphone held by an object or person of interest (e.g., a tennis player), as well as record video from a remote smartphone camera at the same time. The capture setup is presented in Figure 2. Regarding recording motion sensor data, get the accelerometer and gyroscope data of the smartphone. This gives, for example, vertical acceleration, i.e. acceleration in the direction of gravity.

Regarding the video recording, I want the ball price to be filmed. To do this, it uses the smartphone's high frame rate video recording capability to handle short motions and records video at a rate of 120 fps (or 240 fps if possible). It could have been a bottleneck a few years ago, but the majority of recent smartphones have a high fps mode, and I think this feature will become standard for all future smartphone models as well.

The focus here is on short, strong acceleration motions. Also consider the application to rotational motion and direction. The data processing step has three different variables based on the target effect and the motion sensor used (accelerometer or gyroscope). Therefore, in order to explain the general algorithm of the motion sensor-based approach according to the present embodiment, one general use case per variable will be considered as follows. Take a picture of a jump, a partial video speed re-time of the tennis player hitting the ball (i.e. slow motion at the price, while the rest of the video is normal speed), and camera point selection. From these three variables and motivational examples, below, in the experimental section, we will present a method of generalizing to various applications in the context of automatic filming and video editing.

First, hit detection will be described. A method of processing recorded motion sensor data will be described. From a boxing point of view, we try to find the point when the boxer hits the punching bag. We present a simple but valid approach to detecting price points in a video. The motion sensor-based approach is based on the observation that punches with high acceleration occur while the boxer's fist strikes the punching bag. It detects the moment when you have a punch of great acceleration in response to the impact of your fist on the punching bag.

In the data acquisition section, motion sensor data is recorded from a smartphone held by an object or person of interest, as well as video from a remote smartphone camera. From a boxing point of view, a smartphone attached to a sandbag records motion sensor data. The capture setup is shown in FIG. 7. Regarding the motion sensor data recording, get the accelerometer and gyroscope data of the smartphone. In boxing, information from the accelerometer and gyroscope is used to calculate the magnitude of the acceleration that occurs along a plane perpendicular to the gravity vector. Basically, you get the value of the acceleration that occurs along the direction of the punch. Now that the data processing and price detection have been described, a method of processing the recorded motion sensor data will now be described. Trying to find out when the boxer hits the punching bag. We present a simple but valid approach to detecting price points in a video.

Jump detection will be described. As explained when reviewing relevant studies, triggering the camera automatically at the correct time to take a jump picture (e.g., via vision analysis or motion sensor) is inevitable, variable, and proactive. It is not satisfied by some reasons such as unknown data processing and delay in camera operation. To overcome these limitations, a motion sensor-based approach records a video of a jumping person and automatically selects the optimal video frame from the motion sensor data. Immediately after the jump, the app of the motion sensor-based approach method according to the present embodiment analyzes the motion sensor data directly on the smartphone in order to detect the jump time. Then, the detected time is wirelessly transmitted to the camera phone, and a video frame corresponding to the time is automatically selected. Finally, the result of this video frame is displayed on the user's smartphone. The whole process is carried out in a completely automatic way, and the resulting picture is displayed approximately 2 seconds (total processing time) after stopping video recording.

A motion sensor-based approach to detecting the jump time is based on the observation that two large accelerations occur during the jump. As shown in Fig. 5, one is at the time of jumping and the other is at the time of landing. The motion sensor-based approach uses these two large accelerations in a robust and simple way to detect the start and end of a jump, and considers the middle time of the jump as the view of the highest position. In theory (as discussed later) the assumption that the jump is “symmetrical” with respect to the time at the highest position, eg, the jump starts and ends at the same position (eg, the ground). In practice, it will be shown that the motion sensor-based approach gives visually pleasing results even when this assumption is completely invalid.

만큼 떨어진 곳까지의 모든 피크들을 무시한다. 이후, 두 번째로 높은 나머진 피크에 대해 절차를 반복한다. 모든 실험에서,

라고 설정한다(점프 지속 시간은 일반적으로 200ms 및 700ms 사이에서 변한다). 이는 매우 보수적인 값이며 전반적으로 이 과정은 실험에서 얻는 300개의 점프 중 대다수(>90%)에 대해 만족스러운 결과를 제공했다. 따라서, 이 두 개의 피크의 시각

및

를 얻는다. 이후, 가장 높은 위치의 시각

를 점프의 중간 시각, 즉

로 계산한다. Now, how to detect the two acceleration peaks will be described. We analyzed more than 300 jumps and observed that the acceleration values at the time of jumping and landing were not so large, and in fact, the acceleration during the jump was the largest. Therefore, it aims to detect the two largest acceleration peaks in the motion sensor data. To find these two acceleration peaks, we simply extract the two local maximums from the (vertical) acceleration data. Specifically, extract the highest peak from the acceleration data and the minimum peak distance from that peak.

Ignore all peaks up to the distance by. Then, the procedure is repeated for the second highest remaining peak. In all experiments,

(Jump duration typically varies between 200ms and 700ms). This is a very conservative value and overall this process gave satisfactory results for the majority (>90%) of the 300 jumps obtained in the experiment. Therefore, the time of these two peaks

And

Get Afterwards, the time of the highest position

The middle time of the jump, i.e.

Calculate as.

Since it has the time of the highest jump position, it is necessary to simply search for a video frame corresponding to that time. This step requires device synchronization, which will be described below.

Ignoring the air resistance, the jumping person's motion during the jump state is a trajectory. That is, people are only affected by gravity. Hereinafter, it is assumed that the jump is "symmetric". The jump trajectory is symmetrical with respect to the view of the highest position. For example, a jump starts and ends at the same height (eg, the ground). In that case, the highest position is approximately reached in the middle of the jump. For a particular jump, the height of the start and end of the jump may be different, and there may be additional forces/torques that make the jump asymmetrical. Nevertheless, the motion sensor-based approach still works for a wide variety of jump styles, as shown in the experiment, and provides visually appealing results with varying heights. In a symmetrical jump, the time to reach the highest position corresponds to the middle time of the jump. Therefore, the goal is to find the start (leap) and end (landing) time of the jump, which will be described below. It will be shown that by using the fact that data on the overall jump is available, these views can be easily obtained without integration.

Orientation detection will be described. The automatic camera selection approach is based on the smartphone's rotation matrix data acquired by the accelerometer and magnetic field sensor. The rotation matrix converts the vectors in the smartphone's coordinate system into the world's coordinate system, where the x-axis is defined as the cross-product of the y-axis and z-axis (which is in contact with the ground at the device's current position, It points approximately east), the y-axis touches the ground at the device's current position, points to the magnetic north pole, and the z-axis points to the sky and is perpendicular to the ground. By using the rotation matrix data, the z-axis vector of the smartphone (the vector comes out of the device's screen and is vertical) and the direction vector (the opposite of the z-axis vector, comes out of the device's camera and is perpendicular to its back side) is converted into the world coordinate system. Can be calculated from. This makes it possible to compare the z-axis and direction vectors of several nearby smartphones.

6 is a diagram for explaining automatic camera selection in setting a plurality of cameras according to an exemplary embodiment.

,

,

등), 관심 있는 물체를 위한 하나의 스마트폰(예를 들어, 사람, 인형 등) 및 데이터 기록을 위해 개발된 어플리케이션을 구비한다. 카메라 장치들은 다른 각도에서 물체를 보고 있으며, 물체는 비디오 내내 방향을 바꾼다. 카메라들이 다른 시간에(일부는 빨리, 일부는 늦게) 기록을 시작하기 때문에, 정확히 같은 시각(time)에 시작하고 같은 시간(duration)동안 되도록 만들기 위해 촬영 이후 개별적인 비디오들을 편집(cut)해야 할 필요가 있다. 이러한 목적 및 또 다른 목적으로, 장치들 간의 동기화가 요구된다. 따라서, 공공 시간 서버(public time server)를 사용하기로 결정했지만, 장치들 간의 블루투스 또는 와이파이 P2P 연결 또한 잘 동작한다. 관심 있는 물체에 부착된 장치(

)의 방향은 물체(

)의 방향과 동일하지 않기 때문에, 매우 유사하더라도(예를 들어, 주머니 내부의 스마트폰은 사람의 실제 방향과 스마트폰의 방향 벡터 간의 작은 각도가 있다), 실제 촬영하기 전에 카메라 스마트폰의 회전 행렬들의 교정 값(calibration values)이 기록되어야 한다. 이는 다음과 같은 방법으로 수행된다. 관심 있는 물체는 첫 번째 카메라로 회전하고 어플리케이션은 교정 값들 및 타임 스탬프(timestamp)를 기록한다. 그 후, 물체는 두 번째 카메라로 회전하고 그렇게 반복한다. 교정 값들은 교정 각들(

,

,

등)을 계산하기 위하여 사용되며, 이는 정확한 카메라 시점을 인지하는 것을 도와줄 것이다. 비디오를 촬영하는 동안, 모든 스마트폰들(카메라 및 물체 장치)은 타임 스탬프들과 함께 회전 행렬 데이터를 기록한다. 기록 이후, 이 데이터는 세계 좌표계에 대해 모든 폰들의 z축 및 방향 벡터를 계산하는 데 사용된다. 카메라들의 z축 벡터들(

,

,

)은 시간에 따라 물체의 폰 방향 벡터(

)와 비교되고, 이에 따라서, 정확한 카메라 시점이 선택된다. 더 정확하게, 프레임들이 다음과 같은 경험(heuristics)에 따라 선택된다. 매 프레임 및 매 카메라 장치에 대해, 카메라 z축 벡터 및 물체의 장치 방향 벡터 간의 각도는 (내적(dot product)을 통해) 계산되며, 다음 식과 같이 표현될 수 있다.6, two or more camera smartphones (

,

,

Etc.), one smartphone for the object of interest (for example, a person, a doll, etc.), and an application developed for data recording. Camera devices are looking at the object from different angles, and the object changes direction throughout the video. Since cameras start recording at different times (some early, some late), it is necessary to cut individual videos after shooting to make them start at exactly the same time and for the same duration. There is. For this and another purpose, synchronization between devices is required. Therefore, although it was decided to use a public time server, Bluetooth or Wi-Fi P2P connections between devices also work well. Devices attached to objects of interest (

The direction of the object (

), because it is not the same as the orientation of the camera, although very similar (for example, the smartphone inside the pocket has a small angle between the person's actual orientation and the smartphone's orientation vector), the rotation matrix of the camera smartphone before the actual shooting The calibration values of these should be recorded. This is done in the following way. The object of interest is rotated with the first camera and the application records the calibration values and timestamp. After that, the object rotates with the second camera and repeats. The calibration values are the calibration angles (

,

,

Etc.), which will help to recognize the correct camera point of view. During video shooting, all smartphones (camera and object device) record rotation matrix data along with time stamps. After recording, this data is used to calculate the z-axis and direction vectors of all pawns with respect to the world coordinate system. The z-axis vectors of the cameras (

,

,

) Is the object's pawn direction vector over time (

), and accordingly, the correct camera viewpoint is selected. More precisely, frames are selected according to the following heuristics. For every frame and every camera device, the angle between the camera z-axis vector and the device direction vector of the object is calculated (through a dot product), and can be expressed as the following equation.

[Equation 1]

, 즉 교정 각)와 비교되며, 이 두 각도 간의 가장 작은 절대 값 차이를 갖는 카메라가 선택된다. 즉, 다음 식과 같이 표현될 수 있다.It is important to note that the angle between the z-axis vector of the camera and the device direction vector of the object is defined as a directed counterclockwise angle from the vector of the camera to the direction vector of the object device. After that, the angle at the time of calibration (

, That is, the calibration angle), and the camera with the smallest absolute difference between these two angles is selected. That is, it can be expressed as the following equation.

[Equation 2]

The following describes device synchronization.

Since you have the time of the motion of interest in the motion sensor data timeline (ie, the punch/price, the highest jump position, the time point), you need to find the corresponding time in the video timeline. In practice, motion sensor data and video are not temporally aligned because motion and camera phones may have different clock times. Therefore, it is necessary to synchronize the phone before searching for a video frame.

As explained above, the phones' clocks are not the same (ie, time zones, network settings, manual settings, etc.) so the motion sensor data and video are not temporally aligned. That's why you need to sync your phone. The smartphones are temporally synchronized through a wireless connection to a reference time (one of the phones). We tested two methods of providing satisfactory synchronization results. The first method is to send a reference time (ie, any one of the devices called a reference phone) over a Bluetooth network or local Wi-Fi between devices. The second method is to connect both devices to a public time server through an Internet connection to obtain a common reference time. Thereafter, given the reference time, one method is to change the clock of the mobile phone to the reference time, which can be inconvenient to the user (ie, time zone, user preference, etc.). Instead, it measures and stores the difference between the reference time and the phone's internal clock. This synchronization process is performed only once and is fully automatic when the app is launched.

를 모션 폰의 시계에서 카메라 폰의 시계로 변환할 수 있어서, 점프 시간에 대응하는 비디오 프레임을 자동적으로 검색할 수 있다. 이제 휴대폰들이 동기화되었다고 가정하면, 시간

를 (앱에서 설정된) 비디오 프레임 속도를 알려주는 프레임 인덱스 y로 변환할 수 있으며, 그 시간 인덱스의 비디오 프레임을 반환할 수 있다. 계산된 프레임 인덱스가 정수가 아닌 경우에, 시간적으로 가장 가까운 비디오 프레임을 선택한다. 대표적인 결과는 도 5의 (b)에 도시된 바와 같다.Thanks to this synchronization, now

Can be converted from the motion phone's watch to the camera phone's watch, so that the video frame corresponding to the jump time can be automatically searched. Now, assuming that the phones are synchronized, the time

Can be converted to a frame index y indicating the video frame rate (set in the app), and a video frame at that temporal index can be returned. If the calculated frame index is not an integer, the temporally closest video frame is selected. Representative results are as shown in Figure 5 (b).

Based on the general algorithm of the motion sensor-based approach according to the present embodiment, it can now be extended to multiple motion sensors.

7 is a diagram for describing a motion sensor-based approach to a plurality of motion sensors in the context of a group jump photograph according to an exemplary embodiment.

As shown in Fig. 7, consider the case of several people, each wearing a motion phone and being photographed simultaneously by one camera phone. 7(a) shows the data acquisition setup, where two jumping people wear motion phones, and their jumping video is recorded on the camera phone. (b), (c) and (d) may not have an image where the two people are in the highest position, since two people may not jump at the same time. Accordingly, according to an embodiment, as shown in (b) and (c), an image in which each jumping person is at the highest position may be automatically searched. Thereafter, as shown in (c), these two images can be combined to produce the final result in which the two people are in the highest position.

In terms of the user interface, users simply need to select their phone's mode (motion sensor).

Based on the general approach of the motion sensor-based approach according to the present embodiment described above, it can be extended to a plurality of cameras.

8 is a diagram illustrating an arrangement of a smartphone for data capture for a bullet time effect according to an exemplary embodiment.

As shown in Fig. 8, consider the case of one person wearing a motion phone and being photographed by multiple cameras at the same time. This setting allows the creation of the bullet time effect known by the Movie Matrix in a fully automatic way. Compared to the complex commercial camera array system, the motion sensor-based approach is a low-cost, low-weight camera array that is easy to use by everyday users using smartphones that are ubiquitous. Previously, the use of smartphone cameras was considered, but it was necessary to manually select the time to “freeze”. In contrast, the motion sensor-based approach according to this embodiment can operate in a fully automatic manner.

For the motion sensor-based approach according to this embodiment, the camera order must be known (the leftmost one, the second one on the left, etc.). This can be resolved manually. To get the camera sequence automatically, you can use the gyroscope data (yaw angle).

9 is a diagram illustrating an example of a bullet time point according to an embodiment.

Referring to FIG. 9, as an example of a bullet effect viewpoint automatically photographed by a motion sensor-based approach method using a smartphone system, the viewpoints are cropped for better display.

The user can simply swipe the smartphone screen and browse through the captured bullet time point as a slide show.

10 is a diagram illustrating an example of gyro-based browsing according to an embodiment.

In addition, as shown in FIG. 10, a gyroscope-based approach can be implemented in which the user can naturally rotate the smartphone. That is, as an example of gyro-based browsing through photographed bullet effect viewpoints, the user can intuitively rotate the smartphone.

When capturing, the orientation of each camera obtained from the smartphone's gyroscope is stored. After that, when the user sees viewpoints during display, the gyroscope value of the smartphone is read and automatically switches to the nearest bullet viewpoint.

According to embodiments, by applying a motion effect (eg, some slow motion) at a precise point in time, photos and videos are automatically operated using a smartphone that automatically operates without performing visual tracking or video analysis. A motion sensor-based approach and apparatus for shooting and editing can be provided.

As described above, according to the embodiments, automatic camera control, photo/video capture, frame selection, video editing (e.g. local slow motion), camera selection (in multi-camera settings), joint audio-video editing (e.g., based on music). One video retiming), etc. are possible. And, the approach according to the embodiments is executed completely automatically. Automatically recording data, processing, and returning results, is fully automatic and can be run in seconds. In addition, the approach according to the embodiments is intuitive, software-only, and performs simple calculations. The technology according to these embodiments can be used with major social network apps such as Instagram, Facebook and Snapchat for photo/video special effects (e.g. auto jump photos, auto slow motion, optimal frame selection, sound/music effects, logo overlay). It can also be integrated with.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to operate as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (16)

A synchronization step in which the first terminal and the second terminal are connected through wireless communication;
A data acquisition step of recording a video with a camera of the first terminal; And
Data processing step of receiving recorded motion sensor data from the second terminal possessed by a person or object to be photographed, and analyzing and detecting the motion sensor data
Including a motion sensor-based approach. The method of claim 1,
The synchronization step,
Synchronizing the first terminal and the second terminal to temporally align the timeline of the motion sensor data with the timeline of the video
A motion sensor-based approach, characterized in that. The method of claim 1,
The data processing step,
A video frame or segment corresponding to the motion time by receiving the motion time of the motion sensor data at the moment when acceleration of more than a threshold value occurs from the second terminal or the moment the direction of the second terminal coincides with the target direction Steps to automatically select
Including a motion sensor-based approach. The method of claim 3,
The data processing step,
Setting the threshold value through adhoc processing technology
Further comprising a motion sensor-based approach. The method of claim 3,
The data processing step,
Providing at least one video effect of slow motion, camera selection, text superimposition, audio effect, and music to a video frame or segment corresponding to the motion time
Further comprising a motion sensor-based approach. The method of claim 1,
The data processing step,
Detecting two times having the highest vertical acceleration acquired from the second terminal possessed by the person to be photographed in order to automatically calculate the time of the jump position; And
Recognizing the two detected times as jumping and landing, recognizing an intermediate time between the two times as a jump time, and returning a corresponding video frame
Including a motion sensor-based approach. The method of claim 1,
The data processing step,
Detecting a peak of acceleration acquired by the second terminal of a person or object to be photographed in order to automatically calculate a hit time; And
Recognizing the detected acceleration peak as a hit time, and returning a corresponding video frame
Including a motion sensor-based approach. The method according to claim 6 or 7,
The second terminal,
Acquiring accelerometer or gyroscope data
A motion sensor-based approach, characterized in that. The method of claim 1,
The first terminal and the second terminal,
It is a smart phone, and automatically edits the video after shooting using the smart phone.
A motion sensor-based approach, characterized in that. A synchronization unit to which the first terminal and the second terminal are connected through wireless communication;
A data acquisition unit for recording video with a camera of the first terminal; And
A data processing unit that receives recorded motion sensor data from the second terminal possessed by a person or object to be photographed, and analyzes and detects the motion sensor data
Containing, motion sensor-based access device. The method of claim 10,
The synchronization unit,
Synchronizing the first terminal and the second terminal to temporally align the timeline of the motion sensor data with the timeline of the video
A motion sensor-based approach device, characterized in that. The method of claim 10,
The data processing unit,
A video frame or segment corresponding to the motion time by receiving the motion time of the motion sensor data at the moment when acceleration of more than a threshold value occurs from the second terminal or the moment the direction of the second terminal coincides with the target direction To automatically select
A motion sensor-based approach device, characterized in that. The method of claim 12,
The data processing unit,
Providing at least one video effect of slow motion, camera selection, text superimposition, audio effect, and music to a video frame or segment corresponding to the motion time
A motion sensor-based approach device, characterized in that. The method of claim 10,
The data processing unit,
In order to automatically calculate the time of the jump position, two times with the highest vertical acceleration acquired from the second terminal possessed by the person to be photographed are detected, and the detected two times are recognized as jumping and landing. , Recognizing an intermediate time between the two times as a jump time and returning a corresponding video frame
A motion sensor-based approach device, characterized in that. The method of claim 10,
The data processing unit,
In order to automatically calculate the hit time, the peak of the acceleration acquired by the second terminal possessed by the person or object to be photographed is detected, and the detected peak of the acceleration is recognized as the hit time, Returning the corresponding video frame
A motion sensor-based approach device, characterized in that. The method of claim 10,
The first terminal and the second terminal,
It is a smart phone, and automatically edits the video after shooting using the smart phone.
A motion sensor-based approach device, characterized in that.

Images