KR20220087359A - Method and electronic device for producing summary of video - Google Patents

Abstract

Disclosed is a method for an electronic device 100 to generate a video summary. The method includes receiving a video comprising a plurality of frames. The method also includes determining a viewpoint of a user viewing the video. The method also includes determining whether at least one region of interest (ROI) of the user is available in the video based on the user's viewpoint. The method also includes identifying a set of frames from the plurality of frames including the at least one ROI in response to determining that the at least one ROI is available in the video. The method also includes generating a video summary based on the identified set of frames.

Description

METHOD AND ELECTRONIC DEVICE FOR PRODUCING SUMMARY OF VIDEO

The disclosed invention relates to a system for generating a video summary, and more particularly, to a method and an electronic device for generating a video summary based on a viewpoint of a user viewing the video.

Many existing methods and systems have been proposed for generating video summaries. In existing methods and systems, video summaries are generated based on at least one of geometric interpretation to extract regions of interest for key frame identification, camera angles for regions of interest, face color histograms, object information, and intelligent video thumbnail selection and generation. However, these existing methods and systems may have advantages and disadvantages in terms of power consumption, memory usage, robustness, reliability, integrity, operational dependence, time, cost, complexity, design, hardware components used, size, etc. . Also, capturing a viewpoint from the user's point of view is difficult, and current neural networks cannot simulate such things as current neural networks retain historical information. In addition, current deep learning systems such as Long Short-Term Memory (LSTM)/Gated Recurrent Unit (GRU) are limited in capturing and generating summaries in one video.

Accordingly, it would be desirable to address the aforementioned or other drawbacks, or at least provide a useful alternative.

One aspect of the disclosed invention is to provide a method and an electronic device for generating a summary of a video based on a viewpoint of a user watching the video.

One aspect of the disclosed invention is to provide a frame selection technique according to input excitation per frame or frame sequence. "Input excitation" is a weighted parameter that is compared while selecting a keyframe from a view point.

One aspect of the disclosed invention is intended to provide for determining a change in the user's viewpoint (view point) of the user. Thus, it is possible to dynamically generate video summaries in a cost-effective manner. Viewpoints can aid in probabilistic decisions to generate multiple video summaries of one video itself.

One aspect of the disclosed subject matter seeks to provide for generating a video summary by capturing environmental input, a user's thoughts, positive or negative ratings or reviews.

Accordingly, an embodiment of the disclosed invention discloses a method for an electronic device to generate a video summary. The method may include receiving, by the electronic device, a video including a plurality of frames. Also, the method may include determining, by the electronic device, at least one viewpoint of a user viewing the video. The viewpoint may include at least one of the user's subjective viewpoint, the user's objective viewpoint, and the user's physical viewpoint. Also, the method may include, by the electronic device, determining whether at least one region of interest (ROI) of the user is available in the video based on the at least one viewpoint of the user. Further, the method may include, in response to the electronic device determining that the at least one ROI is available in the video, identifying a set of frames from the plurality of frames including the at least one ROI. Further, the method may include generating, by the electronic device, a video summary based on the identified set of frames. Further, the method may include the electronic device storing a summary of the video.

According to an embodiment, when the electronic device determines the subjective viewpoint of the user watching the video, the electronic device acquires a plurality of subjective parameters related to the user. The plurality of subjective parameters may include at least one of the user's occupation, the user's age, the user's preferences, an event related to the user, and the user's activity on the at least one social networking site. Also, the method may include determining, by the electronic device, subjective context information of the user based on a plurality of subjective parameters related to the user. Also, the method may include determining, by the electronic device, a subjective viewpoint of the user based on the user's subjective context information.

According to an embodiment, when the electronic device determines the objective viewpoint of the user watching the video, the electronic device acquires a plurality of objective parameters related to the user, and the electronic device responds to the plurality of objective parameters related to the user. determining the objective context information of the user based on the user's objective context information, and determining, by the electronic device, the objective point of view of the user based on the objective context information of the user, wherein the plurality of objective parameters include the user's past history, the user's current goal, and the user. may include at least one of the additional goals of

According to an embodiment, when the electronic device determines the physical viewpoint of the user watching the video, the electronic device acquires a plurality of physical parameters, and the electronic device determines the user based on the plurality of physical parameters related to the user. and determining, by the electronic device, physical context information of the user, and determining, by the electronic device, a physical viewpoint of the user based on the user's physical context information, wherein the plurality of physical parameters include an angle of a camera associated with the user, a user's location, and a user's surroundings. may include at least one of an ambient light condition of , a weather condition around the user, and a privacy preference of the user.

According to an embodiment, wherein the electronic device identifies the set of frames from the plurality of frames including the at least one ROI, the electronic device selects the plurality of frames from the video based on a plurality of excitation parameters associated with each frame. determine an excitation level of each frame, the electronic device extracts at least one of an audio parameter and a text parameter of each frame, and the electronic device determines an excitation level of each frame and a video based on at least one of an audio parameter and a text parameter determining relative context information of each frame from a plurality of frames of The excitation parameter may include at least one of a speed of the ROI in each frame, an intensity of the ROI in each frame, a frequency of appearance of the ROI in each frame, and a reproduction period of each frame.

According to one embodiment, the electronic device generating a summary of the video based on the identified set of frames comprises: determining, by the electronic device, a weight for each frame from the identified set of frames based on the user's ROI and viewpoint; , the electronic device sequencing each frame from the identified frame set based on the weight determined for each frame, and the electronic device merging the sequenced frame set to generate a video summary.

According to an embodiment, when the electronic device determines the weight for each frame from the identified frame set based on the user's ROI and the viewpoint, the electronic device determines the weight for each frame from the user's at least one viewpoint and the identified plurality of frames. obtaining a perspective angle of each frame from a relationship parameter between each frame of , and a plurality of identified frames, and determining, by the electronic device, a weight for the identified frame based on the obtained relationship parameter and the relation parameter may identify at least one of an angle of the video based on the at least one viewpoint of the user and a perspective view of a scene in the identified frame.

According to an embodiment, when the electronic device identifies the frame set from the plurality of frames including the at least one ROI, the electronic device determines an absolute completeness score of the video, and the electronic device determines the absolute completeness score of the video. determining absolute frame excitation information of the video based on the absolute completeness score, the electronic device detecting co-reference information of the video based on the absolute frame excitation information, and and determining, by the electronic device, a sequence excitation level of the video based on the joint reference information.

According to an embodiment, when the electronic device determines the absolute frame excitation information of the video based on the absolute completeness score, the electronic device determines the speed of the ROI in each frame, the intensity of the ROI in each frame, and the intensity of the ROI in each frame. obtain the frequency of appearance of the ROI, and the reproduction period of each frame, and the electronic device acquires the speed of the ROI in each frame acquired, the intensity of the ROI in each frame acquired, the appearance of the ROI in each acquired frame determining absolute frame excitation information of the video based on the frequency, and the obtained playing period of each frame.

According to one embodiment, the absolute completeness score is obtained by obtaining absolute frame information associated with the video, obtaining a completeness threshold associated with the video, and comparing the obtained absolute frame information associated with the video to an obtained completeness threshold associated with the video, can be decided.

According to one embodiment, the absolute frame excitation information may be configured to drive a relative frame excitation associated with the frame set to sequence the frame set. Absolute frame excitation information can be captured independently and matched with segments of reference excitation in context information. Relative frame excitation can be defined by excitation coverage per frame in order. A reference frame excitation may be input, and the frame sequence may be adjusted accordingly to obtain a reference excitation level.

According to an embodiment, the joint reference information may be configured to maintain a level of sequence excitation associated with the set of frames, wherein the joint reference information may include audio usage associated with the set of frames and semantic similarities associated with the set of frames. and determining the joint reference information based on at least one of an obtained audio usage associated with the frame set and an obtained semantic similarity associated with the frame set.

According to one embodiment, the sequence excitation level may be configured to map a similarity associated with a set of frames.

Accordingly, an embodiment of the present invention discloses an electronic device for generating a video summary. The electronic device may include a memory and a video summary generation controller coupled to the processor. The viewpoint-based video summary generation controller may be configured to receive a video including a plurality of frames and determine at least one viewpoint of a user viewing the video. The viewpoint may include at least one of the user's subjective viewpoint, the user's objective viewpoint, and the user's physical viewpoint. The viewpoint-based video summary generation controller may be configured to determine whether at least one ROI of the user is available in the video based on the at least one viewpoint of the user. The view-based video summary generation controller may be configured to identify a set of frames from the plurality of frames comprising the at least one ROI in response to the at least one ROI being available in the video. The viewpoint-based video summary generation controller may be configured to generate a video summary based on the identified set of frames. The viewpoint-based video summary generation controller may be configured to store the video summary.

These and other aspects of embodiments of the invention will be fully appreciated and understood when considered in conjunction with the following description and accompanying drawings. It is to be understood, however, that the following description is presented by way of illustration and not limitation, while indicating the preferred embodiments and numerous specific details thereof. Many changes and modifications can be made within the scope of the embodiments of the present invention, and the embodiments herein include all such modifications.

According to an aspect of the disclosed invention, it is possible to provide a method and an electronic device for generating a summary of a video based on a viewpoint of a user watching the video. As a result, the user experience is improved.

According to one aspect of the disclosed invention, it is possible to provide a frame selection technique according to an input excitation per frame or frame sequence. "Input excitation" is a weighted parameter that is compared while selecting a keyframe from a view point.

According to one aspect of the disclosed invention, it may be provided to determine a change in the user's view point of the user. Thus, it is possible to dynamically generate video summaries in a cost-effective manner. Viewpoints can aid in probabilistic decisions to generate multiple video summaries of one video itself.

According to one aspect of the disclosed subject matter, it may be possible to provide for generating a video summary by capturing environmental input, a user's thoughts, positive or negative ratings or reviews.

1 illustrates various hardware components of an electronic device for generating a video summary according to an embodiment.
2 is a flow diagram illustrating a method of generating a video summary according to one embodiment.
3 is an exemplary diagram in which an electronic device generates a video summary based on an objective viewpoint of a user, according to an embodiment.
4 is an exemplary diagram in which an electronic device generates a video summary based on a physical viewpoint of a user, according to an embodiment.
5 is an exemplary diagram in which an electronic device generates a video summary based on a subjective viewpoint of a user, according to an embodiment.
6A to 6C are exemplary views summarizing an entire soccer match summary at different points in time, according to an embodiment.
7A-7C are exemplary diagrams summarizing a cake cutting ceremony from multiple frames of a video, according to an embodiment.
8A is an exemplary diagram illustrating key frame selection according to an embodiment.
FIG. 8B is an example graph in which an excitation score and a completeness score are calculated based on key frames selected in conjunction with FIG. 8A according to one embodiment.
9 is an exemplary diagram illustrating frame sequencing according to an embodiment.
10 is an exemplary diagram illustrating coordination frame sequencing based on a reference excitation frame according to an embodiment.
11 is an exemplary diagram in which a video summary is generated based on multiple views, according to an embodiment.
12 is an exemplary diagram in which a summary of a soccer match is generated based on multiple viewpoints, according to an embodiment.
13 is an exemplary diagram illustrating video frame excitation based on text and audio according to an embodiment.
14 and 15 are exemplary diagrams illustrating absolute completeness of information in a video frame according to an embodiment.
16A is an exemplary diagram illustrating video frame excitation according to one embodiment.
16B is an exemplary diagram illustrating audio frame excitation according to one embodiment.
16C is an exemplary diagram illustrating text frame excitation according to one embodiment.
17 is an exemplary diagram of sequencing a video frame using an input contextual information distribution based on excitation matching in the frame, according to an embodiment.
18 is an exemplary diagram of sequencing a video frame based on excitation matching in the frame associated with FIG. 17 according to an embodiment.
19 is an exemplary diagram illustrating relative frame excitation according to an embodiment.
20 is an exemplary diagram summarizing a single video frame based on relative frame excitation according to one embodiment.
21 is an exemplary diagram in which an electronic device captures viewpoints of a plurality of subjects, according to an embodiment.
22 is an exemplary diagram in which an electronic device captures a narration viewpoint, according to an embodiment.
23 is an exemplary diagram in which an electronic device captures a highlight from a viewpoint, according to an embodiment.
24 is an exemplary diagram in which an electronic device captures progressive excitation, according to an embodiment.
25 and 26 are exemplary diagrams in which an electronic device generates a plurality of video summaries by obtaining multi-view context information, according to an embodiment.
27 is an exemplary diagram in which an electronic device generates a plurality of video summaries by obtaining multi-view context information using a bokeh effect, according to an embodiment.
28 is an exemplary diagram in which an electronic device generates a plurality of video summaries by sequentially using an excitation distribution to obtain multi-view context information, according to an embodiment.

The embodiments herein and their various features and advantageous details are more fully described with reference to the non-limiting embodiments illustrated in the accompanying drawings and detailed in the description below. Descriptions of well-known components and processing techniques are omitted so as not to unnecessarily obscure the embodiments herein. Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments. As used herein, the term "or" refers to non-exclusive unless otherwise specified. The examples used herein are merely intended to facilitate an understanding of how the embodiments herein may be practiced and to enable those skilled in the art to practice the embodiments herein.

As is conventional in the art, an embodiment may be described and illustrated in terms of a described function or block that performs the functions. Such blocks, which may be referred to herein as units or modules, etc., are analog or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, and the like. It is physically implemented by , and may optionally be driven by firmware. For example, the circuit may be implemented in one or more semiconductor chips or on a substrate support such as a printed circuit board or the like. The circuits constituting the block may be implemented by dedicated hardware or a processor (eg, one or more programmed microprocessors and related circuits) or a combination of dedicated hardware for performing some functions of the block and a processor performing other functions of the block. can Each block of an embodiment may be physically separated into two or more interacting and separate blocks without departing from the scope of the present invention. Likewise, blocks of embodiments may be physically combined into more complex blocks without departing from the scope of the present invention.

The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Accordingly, the invention should be construed to extend to any modifications, equivalents and substitutions, particularly in addition to those set forth in the accompanying drawings. Terms such as “first” and “second” may be used to describe various elements in the present specification, but these elements should not be limited by these terms. These terms are generally only used to distinguish one element from another.

Accordingly, according to one embodiment, a method for generating a video summary by an electronic device is provided. The method may include receiving, by the electronic device, a video including a plurality of frames. Also, the method may include determining, by the electronic device, at least one viewpoint of a user viewing the video. The viewpoint may include at least one of the user's subjective viewpoint, the user's objective viewpoint, and the user's physical viewpoint. Also, the method may include, by the electronic device, determining whether at least one region of interest (ROI) of the user is available in the video based on the at least one viewpoint of the user. Further, the method may include, in response to the electronic device determining that the at least one ROI is available in the video, identifying a set of frames from the plurality of frames including the at least one ROI. Further, the method may include generating, by the electronic device, a video summary based on the identified set of frames. Further, the method may include the electronic device storing a summary of the video.

The method may be used for video summarization from a user's point of view. The method may be used to determine a change in the user's viewpoint. Thus, it is possible to dynamically generate video summaries in a cost-effective manner. The method can be used for video summarization from the user's point of view based on key frame sections. The key frame section may be determined by camera settings, object depth context information, subject context information, similar frames and dissimilar frames, and excitation parameters. The method may use reinforcement learning to capture the user's viewpoint to generate a video summary. The method can be used to generate video summaries by capturing environmental inputs, user thoughts, positive or negative ratings or reviews to understand initial bias.

Current deep learning systems such as LSTM/GRU are limited in capturing and generating summaries from one video, but viewpoints can help in probabilistic decisions to generate multiple video summaries for one video itself. Reinforcement learning is one way to solve the current method, it is based on deep generative networks, and can be further extended to reinforcement learning to dynamically capture viewpoints.

The contextual information input in the method may enhance the reinforcement learning model to generate a video summary. A reinforcement learning model can observe the environment by adding new weights to frames based on excitation levels. As a result, video summaries can be generated in an efficient manner.

Referring to the drawings, in particular FIGS. 1-28, a preferred embodiment is shown.

1 illustrates various hardware components of an electronic device 100 for generating a video summary according to an embodiment. The electronic device 100 may include, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, the Internet of Things (IoT), an immersive system, and a virtual reality device. , a foldable device and a flexible device, but is not limited thereto. The electronic device 100 includes a processor 110 , a communicator 120 , a memory 130 , an encoder 140 , a decoder 150 , a display 160 , a camera 170 , and a viewpoint-based video summary generation controller (view). point based video summary generation controller, 180). The processor 110 may be connected to the communicator 120 , the memory 130 , the encoder 140 , the decoder 150 , the display 160 , the camera 170 , and the viewpoint-based video summary generation controller 180 .

The camera 170 may capture video including a plurality of frames. The captured video may be transmitted to the view-based video summary generation controller 180 via the encoder 140 and the decoder 150 . Encoder 140 and decoder 150 may normalize video in a latent vector space using existing techniques. By normalizing parameters such as speed, intensity, frequency, and duration in a multidimensional space, viewpoints and videos can be evaluated on the same scale. In order for the reinforcement learning model to understand that normalizing data between 0 and 1 is an example of a parameter or variable, a parameter such as a minmax scalar can be reduced.

Also, the view-based video summary generation controller 180 may receive a video. The view-based video summary generation controller 180 may be configured to determine a view point of a user who views the video after receiving the video. The viewpoint may be, for example, a subjective viewpoint of the user, an objective viewpoint of the user, and a physical viewpoint of the user, but is not limited thereto.

The subjective viewpoint of the user is obtained by obtaining a plurality of subjective parameters related to the user, determining the user's subjective context information based on the plurality of subjective parameters related to the user, and determining the subjective viewpoint of the user based on the user's subjective context information By doing so, it can be determined The plurality of subjective parameters may include, but are not limited to, for example, the user's occupation, the user's age, the user's preferences, an event related to the user, and the user's activity on a social networking site. A user's activity on social networking sites may include, for example, likes on social networking sites, dislikes on social networking sites, and sharing photos on social networking sites. As an example, the electronic device 100 may generate a video summary based on a user's subjective viewpoint as shown in FIG. 5 . For example, the user of the electronic device 100 may capture a video of a team meeting, and the user may want to summarize the video based on a CEO speech at the team meeting. Based on the input, the electronic device 100 may generate a video based on the CEO speech at the team meeting. For example, in the case of a train accident, the railroad agency may have different viewpoints, the police department may have different viewpoints, the medical personnel may have different viewpoints, and video summaries may be generated according to the different viewpoints. .

The objective viewpoint of the user is obtained by obtaining a plurality of objective parameters related to the user, determining the objective situation information of the user based on the plurality of objective parameters related to the user, and determining the objective viewpoint based on the objective situation information of the user, can be decided. The plurality of objective parameters may include, but are not limited to, for example, the user's past history, the user's current goal, and the user's additional goal. The user's goal may include the user's purpose/motive. The user's past history may include the user's past events. As an example, the electronic device 100 may generate a video summary based on the user's objective viewpoint as shown in FIG. 3 . In FIG. 3 , the user of the electronic device 100 may capture a video of a newborn baby, and the user of the electronic device 100 may summarize the video of the newborn baby. Based on the user's input, the user of the electronic device 100 may generate a video of a newborn baby from 1 month to 12 months.

The physical point of view of the user is to be determined by obtaining a plurality of physical parameters, determining the physical context information of the user based on the plurality of physical parameters related to the user, and determining the physical point of view of the user based on the physical context information of the user. can The plurality of physical parameters may include, but are not limited to, for example, an angle of the camera 170 , a user's location, ambient light conditions around the user, weather conditions around the user, and privacy preferences of the user. As shown in FIG. 4 , the electronic device 100 may generate a video summary based on the user's physical viewpoint in the soccer match.

Based on the user's viewpoint, the viewpoint-based video summary generation controller 180 may be configured to determine whether the user's ROI is available in the video. After determining that an ROI is available in the video, the view-based video summary generation controller 180 may be configured to identify a set of frames from the plurality of frames comprising the ROI.

According to an embodiment, the view-based video summary generation controller 180 may be configured to determine an excitation level of each frame from a plurality of frames of a video based on a plurality of excitation parameters associated with each frame. The plurality of excitation parameters may include a speed of the ROI in each frame, an intensity of the ROI in each frame, a frequency of appearance in each frame, and a playback time of each frame. In addition, the viewpoint-based video summary generation controller 180 extracts audio parameters and text parameters of each frame, and relative context information of each frame from a plurality of frames of the video based on the excitation level and audio parameters and text parameters of each frame. can be configured to determine Also, the view-based video summary generation controller 180 may be configured to identify a frame set from a plurality of frames including the ROI based on the relative context information of each frame. The plurality of excitation parameters may include, but are not limited to, a speed of the ROI in each frame, an intensity of the ROI in each frame, a frequency of appearance in each frame, and a playback time of each frame.

According to an embodiment, the viewpoint-based video summary generation controller 180 determines an absolute completeness score of the video to identify a set of frames from the plurality of frames, and determines absolute frame excitation information of the video based on the absolute completeness score. and to detect joint reference information of the video based on the absolute frame excitation information, determine a sequence excitation level of the video based on the joint reference information, and identify a set of frames from the plurality of frames based on the sequence excitation level of the video. can be configured. Examples relating to absolute completeness scores, absolute frame excitation information, joint reference information, and sequence excitation levels are illustrated in FIGS. 9 and 10 .

The absolute frame excitation information of the video is obtained by obtaining the velocity of the ROI in each frame, the intensity of the ROI in each frame, the frequency of appearance of the ROI in each frame, and the playing time of each frame, It can be determined by determining the absolute frame excitation information of the video based on the speed, the intensity of the ROI in each acquired frame, the frequency of appearance of the ROI in each acquired frame, and the playback time of each acquired frame. As an example, while capturing motion in a scene, velocity may include absolute or relative velocity between subjects. As an example, while capturing subject or object involvement, intensity may include emotion, image heat map, loudness, color heat map, color change, background, animation. As an example, the frequency may include the appearance, repetition of the subject, similar or dissimilar events of the subject. As an example, the duration may include a frame duration for capturing the sequence.

The absolute completeness score may be determined by obtaining absolute frame information associated with the video, obtaining a completeness threshold associated with the video, and comparing the obtained absolute frame information associated with the video to an obtained completeness threshold associated with the video.

The absolute frame excitation information may be configured to drive relative frame excitation associated with the set of frames to sequence the set of frames. Absolute frame excitation information can be captured independently and matched with segments of reference excitation in context information. Relative frame excitation can be defined as excitation coverage per frame in order. A reference frame excitation may be input, and the frame sequence may be adjusted accordingly to obtain a reference excitation level.

The joint reference information may be configured to maintain a sequence excitation level associated with the set of frames. The reference information is configured to obtain semantic similarity associated with a scene and frame set including an audio usage associated with the frame set, and obtain semantic similarity associated with an obtained scene and frame set including an audio usage related to the frame set. It may be determined by determining the joint reference information based on . Sequence excitation levels may be configured to map similarities associated with a set of frames.

View-based video summary generation controller 180 may be configured to generate a video summary based on the identified set of frames. According to an embodiment, the viewpoint-based video summary generation controller 180 may be configured to determine a weight for each frame from the identified frame set based on the user's viewpoint and the ROI. A weight for each frame from the identified frame set is obtained by obtaining a relation parameter between the user's viewpoint and each frame from the identified plurality of frames and a perspective angle of each frame from the identified plurality of frames, may be determined by determining a weight for the identified frame based on the determined relation parameter. The relationship parameter may identify the angle of the video based on the user's viewpoint and a perspective view of the scene in the identified frame. Also, the view-based video summary generation controller 180 may be configured to sequence each frame from the identified frame set based on a weight determined for each frame, and merge the sequenced frame set to generate a video summary.

The view-based video summary generation controller 180 may be configured to store the video summary in the memory 130 . The view-based video summary generation controller 180 may be configured to display the video summary on the display 160 . The display 160 may include, for example, a liquid crystal display (LCD) display and a light emitting diode (LED) display, but is not limited thereto. Display 160 may be implemented using any other touch detection technology. The display 160 may be provided with a mode for generating a video summary. Modes may include, but are not limited to, manual mode, semi-auto mode, and fully automatic mode, for example. According to the selection mode, the electronic device 100 may generate a video summary. The manual mode may operate according to a user's input. The semi-automatic mode may operate based on one or more interests of the user, the user's browsing history, past camera usage, and current status information. The fully automatic mode may operate based on scene analysis and environment analysis by the electronic device 100 .

The viewpoint-based video summary generation controller 180 is physically implemented by analog or digital circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, etc. and may optionally be driven by firmware. The viewpoint-based video summary generation controller 180 may be implemented in one or more semiconductor chips or on a substrate support such as a printed circuit board. The circuits constituting the block may be implemented by dedicated hardware or a processor (eg, one or more programmed microprocessors and related circuits) or a combination of dedicated hardware for performing some functions of the block and a processor performing other functions of the block. can

Also, the memory 130 may store instructions to be executed by the processor 110 . Memory 130 may include a non-volatile storage element. Examples of such non-volatile storage elements may include magnetic hard disks, optical disks, floppy disks, flash memory, or in the form of electrically programmable memories (EPROM) or electrically erasable and programmable memories (EEPROM). Also, memory 130 may be considered a non-transitory storage medium in some examples. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or propagated signal. However, the term “non-transitory” should not be interpreted as meaning that the memory 130 is immovable. In some examples, memory 130 may be configured to store larger amounts of information. In certain instances, the non-transitory storage medium may store data that may change (in random access memory (RAM) or cache) over time.

The processor 110 may be configured to execute instructions stored in the memory 130 and perform various processes. The processor 110 may include one or a plurality of processors. One or more processors include a general-purpose processor such as a central processing unit (CPU) and an application processor (AP), a graphics-only processing unit such as a graphics processing unit (GPU), and a visual processing unit (VPU); and/or an AI-dedicated processor such as a neural processing unit (NPU). The processor 140 may include multiple cores and may be configured to execute instructions stored in the memory 120 .

Communicator 120 may be configured to communicate internally between internal hardware components and external devices via one or more networks. For example, the communicator 120 may include a Bluetooth communicator, a wireless fidelity (Wi-Fi) module, and a Li-Fi module, but is not limited thereto.

As an example, the electronic device 100 may summarize the entire soccer match summary at different time points as shown in FIGS. 6A to 6C . The electronic device 100 may summarize the cake cutting ceremony from a plurality of video frames as shown in FIGS. 7A to 7C .

Although FIG. 1 illustrates various hardware components of the electronic device 100 , it should be understood that other embodiments are not limited thereto. According to another embodiment, the electronic device 100 may include a smaller number of components. In addition, labels or names of components are used for illustrative purposes only and do not limit the scope of the present invention. One or more components may be joined together to perform the same or substantially similar function to create a video summary.

2 is a flow diagram 200 illustrating a method of generating a video summary according to one embodiment. Operations 202-226 may be performed by the viewpoint-based video summary generation controller 180 .

The method may include receiving a video comprising a plurality of frames ( 202 ). The method may include obtaining a plurality of subjective parameters associated with the user ( 206a ). The method may include determining subjective contextual information of the user based on a plurality of subjective parameters associated with the user. The method may include determining the user's subjective viewpoint based on the user's subjective contextual information (208a).

The method may include obtaining a plurality of objective parameters associated with the user ( 204b ). The method may include determining objective contextual information of the user based on a plurality of objective parameters associated with the user ( 206b ). The method may include determining an objective viewpoint of the user based on the objective context information of the user (208b).

The method may include obtaining a plurality of physical parameters (204c). The method may include determining physical context information of the user based on a plurality of physical parameters associated with the user ( 206c ). The method may include determining a physical viewpoint of the user based on the physical context information of the user (208c).

The method may include determining whether an ROI of the user is available in the video based on the user's viewpoint ( 210 ). The method may include determining ( 212 ) an excitation level of each frame from the plurality of video frames based on a plurality of excitation parameters associated with each frame. The method may include extracting 214 audio parameters and text parameters of each frame. The method may include determining relative contextual information of each frame from the plurality of video frames based on an excitation level of each frame and an audio parameter and a text parameter ( 216 ).

The method may include identifying ( 218 ) a set of frames from the plurality of frames comprising the ROI based on the relative context information of each frame. The method may include obtaining ( 220 ) a relationship parameter between the user's viewpoint and each frame from the identified plurality of frames and a perspective angle of each frame from the identified plurality of frames. The method may include determining a weight for the identified frame based on the obtained relationship parameter ( 222 ). The method may include sequencing each frame from the identified set of frames based on the weight determined for each frame ( 224 ). The method may include merging the sequenced set of frames to generate a video summary ( 226 ).

In the method, reinforcement learning can be used to capture the user's point of view and generate a video summary. The method may be used to generate a video summary by capturing environmental input, a user's thoughts, positive or negative ratings or reviews.

Current deep learning systems, such as LSTM/GRU, have limitations in capturing and generating summaries from one video, but viewpoints can help in probabilistic decisions to generate multiple video summaries for one video itself. Reinforcement learning is just one way to solve the current method, it is built on deep networks and can be further extended with reinforcement learning to dynamically capture viewpoints.

In the above method, contextual input may enhance the reinforcement learning model to generate video summaries. Reinforcement learning models can observe the environment and give new weights to frames depending on the excitation level. As a result, it is possible to generate video summaries in an efficient manner.

The various acts, acts, blocks, steps, etc. of flowchart 200 may be performed in the order presented, in a different order, or concurrently. Further, in some embodiments, some of the actions, acts, blocks, steps, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present invention.

8A is an exemplary diagram 800a illustrating key frame selection according to an embodiment. As an example, four frame sets are available in a video, and the electronic device 100 may identify a viewpoint of the video. A keyframe for video generation may be selected based on the viewpoint. An example graph 800b here where a score and a completeness score are calculated based on the selected key frame is shown in FIG. 8B . Referring to FIG. 8B , the absolute frame score may be calculated as a weighted average in consideration of speed, frequency, intensity, and duration. The completeness score may be a distribution parameter such as mean or variance. The completeness score can be compared to the next frame's completeness score so that it can fit into the next sequence. The completeness score may also specify how many deviations in the viewpoint or quality of frame perfection, such as a video frame, may be included, but whether some portions have been truncated to match the viewpoint excitation. Frame completeness can be thought of literally as how much meaning is retained in the resulting frame when one frame has to delegate information to the next in order to align it.

A key frame may be selected based on an excitation evaluation of an absolute frame and a relative evaluation of the context information provided.

Evaluating excitation for absolute frames: Based on the video, we can calculate excitation using 4 parameters. Parameters may include speed, intensity, frequency and duration. The weight can be adjusted to adjust the demand threshold according to the context information by changing the excitation parameter, or to obtain completeness in the qualitative information of the frame. It can support pre- or post-adjustment of frames according to selection criteria, contextual information matching or thresholds.

Relative evaluation of the given context information (text or audio): When excitation parameters are evaluated for video frames, similarly audio and text parameters can be mapped in the latent space in which the video frame resides. This will help to understand the relative contextual information of generalization. Frame selection based on context information may be further derived. Also, the electronic device 100 may change the color of the frame, the background of the frame, and the foreground of the frame to reach or meet the expected excitation level, remove objects from the frame, and replace objects from the frame. .

9 is an exemplary diagram 900 illustrating frame sequencing according to an embodiment. In Fig. 9, weak joint reference frames and strong joint reference frames are identified, and the sequence of frames can be changed from this joint reference to text, audio, contextual information.

10 is an exemplary diagram 1000 illustrating coordination frame sequencing based on a reference excitation frame in accordance with one embodiment. The adjusted frame sequence may be determined by receiving a reference excitation and a currently selected frame excitation. This can be changed for a frame using weights. Depending on the changes here, an adjusted frame sequence may be created. A frame of a specific or variable length may generate (a) an overall excitation level matching the reference excitation or (b) a window wise excitation level. For matching here, (a) weight control can be used to adjust frame characteristics (e.g., speed, intensity, frequency, or duration), and (b) the sequence can be changed if there is a matching frame for the selected sequence. and (c) can be replaced with a new frame in the selection process.

11 and 12 are example diagrams 1100 and 1200 in which video summaries are generated based on multiple views, according to an embodiment.

As an example, a user of electronic device 100 using sensor data to capture user excitation during a movie show may be interested in movies of similar (action 1 or action 2) or dissimilar (action and romantic) genres. can be accepted as input for generating a video summary for Video summaries may be generated based on movies of similar genres, and other video summaries may be created based on movies of dissimilar genres.

As an example, two users of the electronic device 100 may have different views of a movie rating (ie, one user may rate a movie based on fight scenes, and another user may rate a comedy scene in the movie). can be graded based on Based on different perspectives, two different video summaries may be generated.

View point context information can be captured in the latent space in the form of a vector, for example, as a sequence distribution or an overall distribution, such as in the form of mean, variance, etc. A vector of latent space can be regarded as a reference excitation parameter for contextual information. Multiple videos can be created when context information is changed. A slight change in context information can create multiple videos. In another example, a summary of a soccer match may be generated based on multiple viewpoints as shown in FIG. 12 . In a soccer game, a viewpoint may include, but is not limited to, for example, a goal-based viewpoint, a player-based viewpoint, a manager-based viewpoint, a midfield-based viewpoint, a defender-based viewpoint, an attacker-based viewpoint, a penalty viewpoint, and a back number viewpoint.

13 is an exemplary diagram 1300 illustrating video frame excitation based on text and audio according to one embodiment. Video frame excitation may be determined by audio excitation. Speech excitation can be determined by identifying words in a word vector that are close to each other. Speech excitation may be determined by identifying the sentence structure, the phonetic usage portions of the sentence structure, the semantic similarity of the sentence structure, and the cross-reference dependencies of the sentence structure. Similarly, audio excitation can be determined by identifying acoustic characteristics (eg, speed, frequency, etc.) of audio and music inputs.

14 and 15 are exemplary diagrams 1400 and 1500 illustrating absolute completeness of information in a video frame according to an embodiment.

The electronic device 100 may be used to derive the completeness of the video frame using the absolute excitation threshold and weight. The absolute excitation threshold is used for completeness according to the excitation threshold, and can fill the frame sequence with matching excitation. Weights are dynamic weight adjustments to satisfy absolute vizio frame excitation. [Equation 1] can absolutely be used here.

[Equation 1]

Absolute excitation = w1 * velocity + w2 * intensity + w3 * frequency + w4 * duration

(where w1, w2, w3 and w4 are adjustments to reach the absolute excitation threshold).

16A is an exemplary diagram 1600a illustrating video frame excitation according to one embodiment. Video frame excitation can be used to match visual excitation to video frames.

16B is an exemplary diagram 1600b illustrating audio frame excitation in accordance with one embodiment. For audio excitation, similar excitation matching can be obtained for audio of video frame context information to support frame sequences. Any combination of acoustic properties such as speed, rhythm, pitch, beat, time, and weighting for the overall excitation in acoustic parameters can be used.

16C is an exemplary diagram 1600c illustrating text frame excitation in accordance with one embodiment. Text Here, matching media excitation characteristics such as speed at which text can be pronounced, word strength, part-of-speech frequency, text sentence pronunciation time, and similarity weight for dynamic adjustment may be calculated.

17 is an exemplary diagram 1700 of sequencing a video frame using an input contextual information distribution based on excitation matching in the frame, according to one embodiment. The electronic device 100 may reselect frames for excitation matching in order, and may adjust a weight parameter for arranging frames in order. Specifically, the electronic device 100 may change weights at different frame positions for video completeness and sequencing.

18 is an exemplary diagram 1800 of sequencing a video frame based on excitation matching in the frame associated with FIG. 17 according to one embodiment. 18 , strong similarity and weak similarity may be identified, and the electronic device 100 may determine that a frame is jointly referenced according to an excitation sequence.

19 is an exemplary diagram 1900 illustrating relative frame excitation in accordance with one embodiment. 19 , four frames are shown: warmth, a mask, protection from pollution or disease and industrial pollution. A strong similarity here may be that industrial waste is the source of many diseases, and a weak similarity may be that global warming and pollution spread the disease.

20 is an exemplary diagram 2000 summarizing a single video frame based on relative frame excitation in accordance with one embodiment. The electronic device 100 may receive an excitation distribution parameter. Excitation distribution parameters may include, but are not limited to, mean, median, variance, kurtosis, and skewness, for example. The electronic device 100 may summarize the frame based on the excitation distribution parameter. In ③, the electronic device 100 may determine absolute frame excitation. In ④, the electronic device 100 may determine relative frame excitation and strong/weak excitation criteria based on the absolute frame excitation. In ⑤, the electronic device 100 may determine the joint reference frame excitation based on the determined relative frame excitation and strong/weak excitation criteria. Based on the joint reference frame, the electronic device 100 may generate multiple summarized video frames, and may generate a single summarized video frame based on the multiple summarized video frames.

21 is an exemplary diagram 2100 in which the electronic device 100 captures viewpoints of a plurality of subjects, according to an embodiment. In FIG. 21 , a custom characteristic may be defined as a viewpoint. The video summary may be generated based on the driver's point of view. The video summary may be generated based on the eyewitness point of view. The video summary may be generated based on the viewer's point of view. , a video summary may be generated based on a traffic police viewpoint.

22 is an exemplary diagram 2200 in which the electronic device 100 captures a narration viewpoint, according to an embodiment. The user of the electronic device 100 may provide a descriptive input saying "The source of global warming is the expansion of industry". Based on the method, the electronic device 100 may generate a video summary related to "the source of global warming is industrial expansion" in the source image.

23 is an exemplary diagram 2300 in which the electronic device 100 captures a highlight from a viewpoint, according to an embodiment. Based on the method, the electronic device 100 may generate a video summary corresponding to the resumption of the school after closure. Video summaries can include many frames representing student responses, safety measures and political involvement, and parent responses.

24 is an exemplary diagram 2400 in which the electronic device 100 captures progressive excitation, according to an embodiment. The video summary may include multiple frames related to conversation, walking, arguing, fighting, and driving. Multiple frames can be arranged based on the above method.

25 and 26 are exemplary diagrams in which the electronic device 100 generates a plurality of video summaries by obtaining multi-view context information, according to an embodiment.

Based on the video frame, the excitation of the video frame can be calculated using the four best parameters. Parameters may include speed, intensity, frequency and duration. The weight may be adjusted by changing the excitation parameter to (a) adjust the demand threshold according to the context information or (b) obtain completeness in the qualitative information of the frame. This can help with pre- or post-adjustment to a frame depending on selection criteria, context match, or threshold.

Relative evaluation of the given context information (text or audio): As excitation parameters are evaluated for video frames, similarly audio and text parameters can be mapped in the latent space with the video frame. This will help to understand the relative contextual information of generalization. Frame selection based on context information may be further derived.

Also, the electronic device 100 may change the color of the frame, the background of the frame, and the foreground of the frame to reach or meet the expected excitation level, remove objects from the frame, and replace objects from the frame. .

27 is an exemplary diagram 2700 in which the electronic device 100 generates a plurality of video summaries by obtaining multi-view context information using a bokeh effect, according to an embodiment. FIG. 28 is an exemplary diagram 2800 in which the electronic device 100 generates a plurality of video summaries by sequentially using an excitation distribution to obtain multi-view context information, according to an embodiment.

Apart from the frames discussed in the various examples above, the weighted excitation parameters can be adjusted for the selected frame by creating an effect that compensates for the viewpoint threshold. The additional completeness of the frame is balanced through dynamic weight adjustment. In addition, there are visual adjustments such as blur, bokeh, boomerang, slow motion, changing the background, changing the color of the clothes, changing the clothes, segmentation or other image processing techniques, applying AR filters, etc. You can adjust the sound, such as volume, repeating dialogue in frames, adding music effects, and more. You can set video frames such as zoom, camera angle, depth, shutter, ISO, iris control, etc.

Embodiments disclosed in the present invention may be implemented using a network management function executed in at least one hardware device.

The foregoing description of specific embodiments fully reveals the general nature of the embodiments herein so that others may readily modify and/or adapt to various applications, such as the specific embodiments, without departing from the general concept by applying current knowledge. Accordingly, such adaptations and modifications should be understood and intended to be understood within the meaning and scope of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology used herein is for the purpose of description and not limitation. Accordingly, although the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein may be practiced with modifications within the scope of the embodiments described herein.

Claims (26)

A method for an electronic device (100) to generate a video summary, comprising:
receiving, by the electronic device 100, a video including a plurality of frames;
determining, by the electronic device 100, at least one viewpoint of a user viewing the video;
determining, by the electronic device 100, whether at least one region of interest (ROI) of the user is available in the video based on the at least one viewpoint of the user;
in response to determining that the at least one ROI is available in the video, the electronic device (100) identifying a set of frames from the plurality of frames including the at least one ROI;
generating, by the electronic device (100), the video summary based on the identified set of frames; and
and displaying the video summary by the electronic device (100). The method of claim 1, wherein the at least one viewpoint comprises a subjective viewpoint of the user;
The method is
The electronic device 100 acquires a plurality of subjective parameters related to the user, and the plurality of subjective parameters include the user's occupation, the user's age, the user's preference, and an event related to the user, and at least one including at least one of the user's activities on a social networking site;
determining, by the electronic device 100, subjective context information of the user based on the plurality of subjective parameters related to the user; and
and determining, by the electronic device 100, the subjective viewpoint of the user based on the subjective context information of the user. The method of claim 1, wherein the at least one viewpoint comprises an objective viewpoint of the user,
The method is
a process in which the electronic device 100 acquires a plurality of objective parameters related to the user, wherein the plurality of objective parameters include at least one of a past history of the user, a current goal of the user, and an additional goal of the user;
determining, by the electronic device 100, the objective context information of the user based on the plurality of objective parameters related to the user; and
and determining, by the electronic device 100, the objective viewpoint of the user based on the objective situation information of the user. The method of claim 1, wherein the at least one viewpoint comprises a physical viewpoint of the user,
The method is
The electronic device 100 acquires a plurality of physical parameters related to the user, and the plurality of physical parameters include an angle of a camera related to the user, a location of the user, an ambient light condition around the user, and a surrounding light condition around the user. including at least one of a weather condition and a privacy preference of the user;
determining, by the electronic device 100, physical context information of the user based on the plurality of physical parameters related to the user; and
The method further comprising; determining, by the electronic device 100, the physical point of view of the user based on the physical context information of the user. The method of claim 1, wherein the step of identifying the frame set from the plurality of frames including the at least one ROI by the electronic device comprises:
The electronic device 100 determines an excitation level of each frame from the plurality of frames of the video based on a plurality of excitation parameters associated with each of the frames, and the plurality of excitation parameters in each frame a process of including at least one of a speed of the ROI, an intensity of the ROI in each frame, an appearance frequency of the ROI in each frame, and a reproduction period of each frame;
extracting, by the electronic device 100, at least one of an audio parameter and a text parameter of each frame;
determining, by the electronic device 100, relative context of each frame from the plurality of video frames based on the excitation level of each frame and at least one of the audio parameter and the text parameter; and
and identifying, by the electronic device 100, the frame set from the plurality of frames including the at least one ROI based on the relative context information of each frame. The method according to claim 1, wherein the generating of the video summary based on the identified frame set by the electronic device (100) comprises:
determining, by the electronic device 100, a weight for each frame from the identified frame set based on the ROI and the viewpoint of the user;
sequencing, by the electronic device 100, each frame from the identified frame set based on the weight determined for each frame; and
and generating, by the electronic device (100), the video summary by merging the sequenced set of frames. The method of claim 6, wherein the determining of the weight for each frame from the identified frame set based on the ROI and the viewpoint of the user by the electronic device 100 comprises:
The electronic device 100 determines a relation parameter between the at least one viewpoint of the user and each frame from the identified plurality of frames, and a perspective angle of each frame from the identified plurality of frames. obtaining, wherein the relation parameter comprises at least one of an angle of the video based on the at least one viewpoint of the user and a perspective view of a scene in the identified frame; and
and determining, by the electronic device 100, the weight for the identified frame based on the obtained relationship parameter. The method of claim 1, wherein the step of identifying the frame set from the plurality of frames including the at least one ROI by the electronic device comprises:
determining, by the electronic device 100, an absolute completeness score of the video;
determining, by the electronic device 100, absolute frame excitation information of the video based on the absolute completeness score;
detecting, by the electronic device 100, co-reference information of the video based on the absolute frame excitation information; and
and determining, by the electronic device 100, a sequence excitation level of the video based on the joint reference information. The method of claim 8, wherein the determining of the absolute frame excitation information of the video based on the absolute completeness score by the electronic device comprises:
obtaining, by the electronic device 100, the speed of the ROI in each frame, the intensity of the ROI in each frame, the frequency of appearance of the ROI in each frame, and a reproduction period of each frame; and
The electronic device 100 determines the speed of the ROI in each of the acquired frames, the intensity of the ROI in each of the acquired frames, the frequency of appearance of the ROI in each of the acquired frames, and a reproduction period of each of the acquired frames. determining the absolute frame excitation information of the video; According to claim 8, wherein the absolute completeness score,
obtaining absolute frame information related to the video;
obtaining a completeness threshold associated with the video; and
comparing the obtained absolute frame information associated with the video with the obtained completeness threshold associated with the video; 9. The method of claim 8, wherein the absolute frame excitation information is configured to drive relative frame excitation associated with the set of frames for sequencing the set of frames. 9. The method of claim 8, wherein the joint reference information is configured to maintain the sequence excitation level associated with the set of frames, the joint reference information comprising:
obtaining at least one of a scene comprising audio usage associated with the set of frames and semantic similarities associated with the set of frames; and
determining the joint reference information based on at least one of the obtained scene comprising the audio usage associated with the frame set and the obtained semantic similarity associated with the frame set. 9. The method of claim 8, wherein the sequence excitation level is configured to map a similarity associated with the set of frames. An electronic device for generating a video summary, comprising:
display; and
a controller connected to the display; and
The controller is
receive a video comprising a plurality of frames;
determine at least one viewpoint of the user viewing the video;
determine whether at least one ROI of the user is available in the video based on the at least one viewpoint of the user;
identify a set of frames from the plurality of frames including the at least one ROI in response to determining that the at least one ROI is available in the video;
generate the video summary based on the identified set of frames;
and present the video summary on the display. 15. The method of claim 14, wherein the at least one viewpoint comprises a subjective viewpoint of the user,
The controller is
obtain a plurality of subjective parameters associated with the user, the plurality of subjective parameters comprising: an occupation of the user, an age of the user, a preference of the user, and an event associated with the user; at least one of the following activities;
determine subjective context of the user based on the plurality of subjective parameters associated with the user;
and determine the subjective viewpoint of the user based on the subjective context information of the user. The method of claim 14, wherein the at least one viewpoint comprises an objective viewpoint of the user,
The controller is
acquire a plurality of objective parameters associated with the user, wherein the plurality of objective parameters include at least one of a past history of the user, a current goal of the user, and an additional goal of the user;
determine the objective context information of the user based on the plurality of objective parameters associated with the user; and
and determine the objective viewpoint of the user based on the objective context information of the user. The method of claim 14, wherein the at least one viewpoint comprises an objective viewpoint of the user,
The controller is
obtain a plurality of physical parameters associated with the user, the plurality of physical parameters including an angle of a camera 170 associated with the user, a location of the user, an ambient light condition around the user, a weather condition around the user, and at least one of the user's privacy preferences;
determine physical context information of the user based on the plurality of physical parameters associated with the user; and
and determine the physical viewpoint of the user based on the physical context information of the user. 15. The method of claim 14, wherein the controller,
determine an excitation level of each frame from the plurality of frames of the video based on a plurality of excitation parameters associated with each of the frames, the plurality of excitation parameters comprising: a velocity of the ROI in each frame; at least one of an intensity of the ROI, a frequency of appearance of the ROI in each frame, and a reproduction period of each frame;
extracting at least one of an audio parameter and a text parameter of each frame;
determine a relative context of each frame from the plurality of video frames based on the excitation level of each frame and at least one of the audio parameter and the text parameter;
and identify the set of frames from the plurality of frames including the at least one ROI based on the relative context information of each frame. 15. The method of claim 14, wherein the controller,
determine a weight for each frame from the identified set of frames based on the ROI and the viewpoint of the user;
sequence each frame from the identified set of frames based on the weight determined for each frame;
and merge the sequenced set of frames to generate the video summary. The method of claim 19, wherein the controller,
obtain a relationship parameter between the at least one viewpoint of the user and each frame from the identified plurality of frames, and a perspective angle of each frame from the identified plurality of frames, the relationship parameter comprising: at least one of an angle of the video based on the at least one viewpoint of the user and a perspective view of a scene in the identified frame;
and determine the weight for the identified frame based on the obtained relationship parameter. 15. The method of claim 14, wherein the controller,
determining an absolute completeness score of the video;
determine absolute frame excitation information of the video based on the absolute completeness score;
detecting co-reference information of the video based on the absolute frame excitation information;
and determine a sequence excitation level of the video based on the joint reference information. The method of claim 21, wherein the controller,
obtain the speed of the ROI in each frame, the intensity of the ROI in each frame, the frequency of appearance of the ROI in each frame, and a reproduction period of each frame;
Excitation of the absolute frame of the video based on the speed of the ROI in each acquired frame, the intensity of the ROI in each acquired frame, the frequency of appearance of the ROI in each acquired frame, and the playback period of each acquired frame An electronic device configured to determine information. The method of claim 21, wherein the controller,
obtaining absolute frame information related to the video;
obtaining a maturity threshold associated with the video;
and determine the absolute completeness score by comparing the obtained absolute frame information associated with the video to the obtained completeness threshold associated with the video. 22. The electronic device of claim 21, wherein the controller is configured to drive relative frame excitation associated with the set of frames to sequence the set of frames based on the absolute frame excitation information. 22. The method of claim 21, wherein the controller is configured to maintain the sequence excitation level associated with the set of frames based on the joint reference information;
The controller is:
obtain at least one of a scene comprising audio usage associated with the set of frames and a semantic similarity associated with the set of frames;
configured to determine the joint reference information by determining the joint reference information based on at least one of the obtained scene comprising the audio usage associated with the frame set and the obtained semantic similarity associated with the frame set; electronic device. 22. The electronic device of claim 21, wherein the controller is configured to map a similarity associated with the set of frames based on the level of sequence excitation.

Images