Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30244—Camera pose
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
  • G06T2219/20—Indexing scheme for editing of 3D models
  • G06T2219/2016—Rotation, translation, scaling

Definitions

• the present disclosure generally relates to image processing.
• the present disclosure relates to a method and an apparatus for determining an orientation of a video.
• the disclosure provides a method and an apparatus for determining an orientation of a video with reasonable computation load for each orientation-homogeneous part of the video.
• some features are extracted based on the estimated motion (dominant or object-based) of the video scene. From these motion-based features, some frame-based orientation information is computed together with potential changes in orientation. Together with the temporal orientation information, the disclosure also results in an associated segmentation into orientation- homogeneous parts of the video.
• a method for determining an orientation of a video comprises the steps of: estimating a motion of the video; extracting translation-based parameters from the estimated motion of the video; and computing at least one feature giving the evolution of the horizontal translation over time against the evolution of the vertical translation according to the translation based parameters, the feature being used for determining the orientation of the video.
• the method further comprises: extracting rotation-based parameters from the estimated motion of the video; splitting the video into at least one segment separated by the rotations detected according to the rotation-based parameters; and determining the orientation of the video as a function of an integration of said at least one feature over each of said at least one segment.
• an apparatus for determining an orientation of a video comprises a processor configured to: estimating a motion of the video; extracting translation- based parameters from the estimated motion of the video; and computing at least one feature giving the evolution of the horizontal translation over time against the evolution of the vertical translation according to the translation based parameters, the feature being used for determining the orientation of the video.
• a computer program product downloadable from a communication network and/or recorded on a medium readable by computer and/or executable by a processor.
• the computer program product comprises program code instructions for implementing the steps of the method according to one aspect of the disclosure.
• a Non-transitory computer- readable medium comprising a computer program product recorded thereon and capable of being run by a processor.
• the Non-transitory computer- readable medium includes program code instructions for implementing the steps of a method according to one aspect of the disclosure.
• Figure 1 is a flow chart showing a method for determining an orientation of a video according to an embodiment of the disclosure
• Figure 2 is a diagram showing an evolution of the difference of the absolute horizontal and vertical translations over time, while integrated;
• Figure 3 is a diagram showing an example of the evolution of the rotation parameter, while integrated
• Figure 4 is a flow chart showing a method for determining an orientation of a video according to another embodiment of the disclosure.
• Figure 5 is an exemplary diagram showing different possible orientations before and after a clock-wise camera rotation (Case 2 in Table 1 );
• Figure 6 is an exemplary diagram showing different possible orientations before and after a counter-clock-wise camera rotation (Case 3 in Table 1 );
• Figure 7 is block diagram showing a computer device on which the method for determining an orientation of a video according to an embodiment of the disclosure may be implemented.
• An embodiment of the disclosure provides a method for determining an orientation of a video. Next, the method of the embodiment of the present disclosure will be described in details.
• Figure 1 is a flow chart showing a method for determining an orientation of a video according to an embodiment of the disclosure.
• embodiments of the disclosure will be discussed with only distinguishing between 4 orientations of 0°, 90°, -90° and 180° for each frame of the video, i.e. between the two landscape (0°, 180°) and portrait (90°, -90°) orientations.
• the present disclosure therefore will only discuss a classification of the frames into these 4 classes and no further precise orientation angle will be extracted.
• the system will provide a first classification into portrait/landscape without distinguishing between the two possible portrait orientations (90°, -90°) and between the two possible landscape orientations (0°, 180°).
• the disclosure can also be applied to the cases with more complicated orientation classification.
• step S101 it estimates motion of the video. It should be noted that a dominant motion of the video is preferred for the motion estimation in the step S101 . But some object-based motion estimation may be relied on when no dominant motion in the video is to be estimated.
• the dominant motion estimation can be carried out by computing, at each instant, a parametric approximation of the dominant motion of the video.
• Some known solutions can be used for the estimation of dominant motion. For example, the following document disclosed a start-of -the- art technology for this purpose:
• step S1 02 it extracts translation-based parameters and rotation-based parameters from the estimated dominant motion of the video.
• Motion estimators output the estimated motion model from which an estimation of the translations in both horizontal and vertical directions together with an estimation of the rotation can be extracted.
• a motion model can be accessed from the dominant motion estimation, which may contain parameters depending on the motion to be estimated.
• an affine model with 6 parameters can be considered as the matrix below:
• the first two parameters aO and bO respectively correspond to Tx and Ty, the translation values in the x axis and y axis.
• the four remaining parameters provide information on the rotation and zooming motions.
• the translation-based parameters and rotation-based parameters can be integrated over a certain period of time.
• the trapezoidal rule which is a well-known technique for approximating the definite integral, can be applied on a given window.
• the size of the given window can be empirically fixed to, for example, 20 frames. But this size can be adapted according to the context. From the dominant motion estimation, only small translation and rotation values can be accessed from one frame to another. From these values it is difficult to accurately detect rotations and translations.
• the advantage of the integration is to provide a larger view of the motion on a longer period of time.
• step S1 03 it computes at least one feature giving the evolution of the horizontal translation over time against the evolution of the vertical translation according to the translation based parameters.
• a feature can give some clue about whether the video was captured in a portrait or landscape mode. In most of the cases, when the amplitude of the translational component of dominant motion is substantially larger in the horizontal direction than in the vertical one, it is likely that the video was captured in landscape mode, as more panning user than tilting will tend to be used during the capturing of a scene of the video.
• Figure 2 shows an example of evolution of such a feature over time from integrated values of Tx and Ty. Positive values tend to indicate a landscape orientation of the video, while negative values tend to indicate a portrait orientation of the video.
• the above feature may be smoothed over a given sliding time window to improve the accuracy of the analysis.
• step S104 in parallel to the step S103, it splits the video into segments separated by the rotations detected according to the rotation-based parameters, whether integrated or not.
• the rotation-based parameter will in turn give some information on whether the camera was rotated during the capture.
• the segmentation can be carried out through a thresholding of the extracted rotation- based parameters.
• Figure 3 is a diagram showing an example of the evolution of the rotation-based parameters after integration.
• the thresholding will give three segments: segment 1 before the rotation, segment 3 after the rotation and segment 2 which corresponds to the rotation and for which the system will not give any orientation information in the embodiment.
• Figure 3 illustrates an example of splitting the video wherein a simple threshold is applied. Regions above the threshold correspond to clockwise rotations, regions below the opposite of the threshold correspond to counterclockwise rotations, and regions therebetween correspond to regions without rotations.
• the threshold value was fixed to an empirical value of 0.2. But more adaptive can be made according to the video content, for example, something that will only keep values below mean-2sigma.Optionally, a refinement can be added in the detection of the rotation boundaries.
• This refinement will start with a higher threshold that gives markers of potential rotations. Then it goes temporally backward in order to find the rotation start which corresponds to the frame for which the absolute value of the rotation parameter is very low (ie. lower than epsilon value). Next it searches forward for the end of the rotation, which corresponds to the first frame for which the absolute value of the rotation parameter is lower than epsilon.
• step S105 it determines the orientation of the video according to an integration of the at least one feature obtained in the step S103 over each segment obtained by the step S104.
• the orientation of the video is determined by computing the orientation of each segment before and after rotations by integrating Featuretrans, for all frames of the segment, in order to get one representative single value of Featuretrans per segment. In this embodiment, it can simply be the computation of the area below the curve in Figure 2(see hatched regions) over the segments. In another embodiment, an additional threshold T is used on the translation-based parameter. The number of continuous parts in which Featuretrans>T and Featuretrans ⁇ -T is counted, plus the duration of these parts. A simple normalized sum of these two counters will provide a new Integrated' value of Featuretrans.
• an integrated value of Featuretrans over the whole video or over windows of a predefined size is computed. Specifically, if no rotation was detected, then there is only one big segment in the whole video. In this case, the same integration process for Featuretrans can be simply applied as what is done to the segment, which however is done to the whole video. As a variant, if the video is very long, then it can be processed window by window. In this case, the integration will be done once again but over a predefined window size.
• orientation is portrait.
• some additional processing such as for example face detection
• face detection can be applied to distinguish further between the two portrait or the two landscape orientations. It can be appreciated by a person skilled in the art that, by detecting faces in pictures, some information of the most likely orientation of a picture can be obtained, as it is very unlikely that people and therefore faces will be upside down. It is also quite unlikely to have people and therefore faces lying in images than people standing. Such information can be used to further distinguish orientations of a video. No further details will be given in this respect.
• Figure 4 is a flow chart showing a method for determining an orientation of a video according to another embodiment of the disclosure.
• steps S401 -S405 are respectively identical to steps S101 -S105 in Figure 1 .
• a further step S406 is added, which further distinguishes the orientation of the video obtained in the step S405 according to the angle of rotation (for example this angle can be extracted from the rotation-based parameter extracted in the step S402).
• the added step S406 can help distinguishing between the two portraits and the two landscape orientations determined in the step S405.
• step S404 information about whether a rotation took place in the video or not can be obtained (the absolute value of the rotation parameter is above a given threshold).
• the absolute value of the rotation parameter is above a given threshold.
• information on the direction of the rotation can be accessed (if rotation_parameter > 0, then the scene rotates clock-wise, i.e. the camera was turned counter-clock-wise, relatively to the center of the frame, if rotation_parameter ⁇ 0, then the scene rotates counter-clock-wise, i.e. the camera was turned clock-wise, relatively to the center of the frame).
• the final rotation value for the video that corresponds to the rotation may either be an integrated value of the rotation parameter over the segment, or simply: - /+ max_over_segment(abs(rotation_parameter). The sign of this quantity will depend on the sign of the rotation_parameter over current segment.
• orient_before and orient_after mean respectively the orientation of all frames before the rotation and the orientation of all frames after the rotation.
• a rotation parameter is studied in each column. Depending on the sign and value of a parameter, some insight can be obtained on both the fact that a rotation takes place and in this case, in which direction the rotation is taken: clock wise, counter clock wise.
• Figures 5 and 6 illustrate different cases discussed in the Table 1 . From Figures 5 and 6 and from the rotation direction, there are only a few possibilities for the translation values before and after the rotation.
• Figure 5 is an exemplary diagram showing different possible orientations before and after a clock-wise camera rotation (Case 2 in Table 1 ). Only the first two cases are considered in the following description.
• Figure 6 is an exemplary diagram showing different possible orientations before and after a counter-clock-wise camera rotation (Case 3 in Table 1 ). Similarly, only the first two cases are considered in the following description.
• step S406 can help remove this false alarm. Choosing one or the other between these two on-going procedures may depend on a confidence value we may have on both the rotation parameter and the translation parameter.
• the motion of some objects in the scene can be estimated and some translation information of these objects, if any, can be used instead of that of the dominant motion.
• the disclosure advantageously offers the orientation information for all frames of a given video, even when a rotation occurs during the capture. It is based on some light motion-based processing applied on the video and hence does not need an offline learning process.
• the step of motion estimation can be realized in real time and the on-line orientation detection process is applied instantaneously after the motion estimation either over the whole video, or over some predefined time windows.
• the disclosure can be implemented on a video player which for example provides playback of previously captured videos.
• the video player comprises but is not limited videolan on PC, online video websites and smart phones.
• An embodiment of the disclosure provides a corresponding apparatus for determining an orientation of a video.
• FIG. 7 is block diagram showing a computer device 700 on which the method for determining an orientation of a video according to an embodiment of the disclosure may be implemented.
• the computer device 700 can be any kind of suitable computer or device capable of performing calculations, such as a standard Personal Computer (PC).
• the device 700 comprises at least one processor 710, RAM memory 720 and a user interface 730 for interacting with a user.
• the skilled person will appreciate that the illustrated computer is very simplified for reasons of clarity and that a real computer in addition would comprise features such as network connections and persistent storage devices.
• the user interface 730 a user can input/select a video for playback.
• the result of the determined orientation of the video can also be outputted, if needed, to the user by the user interface 730.
• the processor 710 comprises a first unit for estimating a motion of the video.
• the processor 710 further comprises a second unit for extracting translation- based parameters and rotation-based parameters from the estimated motion of the video.
• the processor 710 further comprises a third unit for computing at least one feature giving the evolution of the horizontal translation over time against the evolution of the vertical translation according to the translation based parameters.
• the processor 710 further comprises a fourth unit for splitting the video into segments separated by the rotations detected according to the rotation-based parameters.
• the processor 710 further comprises a fifth unit for determining the orientation of the video as a function of an integration of said at least one feature over each of said segments.
• An embodiment of the disclosure provides a computer program product downloadable from a communication network and/or recorded on a medium readable by computer and/or executable by a processor, comprising program code instructions for implementing the steps of the method described above.
• An embodiment of the disclosure provides a non-transitory computer-readable medium comprising a computer program product recorded thereon and capable of being run by a processor, including program code instructions for implementing the steps of a method described above.
• the present disclosure may be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof.
• the software is preferably implemented as an application program tangibly embodied on a program storage device.
• the application program may be uploaded to, and executed by, a machine comprising any suitable architecture.
• the machine is implemented on a computer platform having hardware such as one or more central processing units (CPU), a random access memory (RAM), and input/output (I/O) interface(s).
• CPU central processing units
• RAM random access memory
• I/O input/output
• the computer platform also includes an operating system and microinstruction code.
• the various processes and functions described herein may either be part of the microinstruction code or part of the application program (or a combination thereof), which is executed via the operating system.
• various other peripheral devices may be connected to the computer platform such as an additional data storage device and a printing device.

Landscapes

• Engineering & Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Multimedia (AREA)
• Image Analysis (AREA)
• Studio Devices (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=50288014

Family Applications (1)

Country Status (6)

Families Citing this family (4)

Family Cites Families (27)

• 2015
  • 2015-02-23 CN CN201580009769.2A patent/CN106030658B/zh not_active Expired - Fee Related
  • 2015-02-23 US US15/122,167 patent/US10147199B2/en not_active Expired - Fee Related
  • 2015-02-23 WO PCT/EP2015/053746 patent/WO2015128294A1/en not_active Ceased
  • 2015-02-23 EP EP15706783.6A patent/EP3111419A1/en not_active Withdrawn
  • 2015-02-23 JP JP2016550835A patent/JP2017512398A/ja active Pending
  • 2015-02-23 KR KR1020167022482A patent/KR20160126985A/ko not_active Withdrawn

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events