TWI535282B - Method and electronic device for generating multiple point of view video - Google Patents

Description

Method and electronic device for generating multi-view video

The present disclosure relates to methods and electronic devices for generating multiple point of view (MPOV) video.

It has various functions to make electronic devices such as smart phones and tablets more mobile and versatile. By using the image capture function of the electronic device, the user will be able to record events by capturing events of their daily life and storing the events as media content in different media formats (eg, photos, videos, audio, etc.). Users often have multiple media content associated with the same event at different viewpoints, and these users may later want to share their media content with different viewpoints via email, social networking, or other means of communication.

However, as such, the user may have to navigate through the media content to manually identify media content related to the event of interest, and these actions can be extremely time consuming. In addition, relevant media content may not necessarily be sorted or synchronized in time, so that the user will have to manually select and re-adjust the relevant media content in order to The content is cataloged as a video compilation or photo album.

Thus, it may be desirable to automatically identify relevant media content for the same event, and to select and combine these related media content for presentation from multiple viewpoints.

The present disclosure proposes methods and electronic devices for generating multi-view (MPOV) video.

According to one of the exemplary embodiments, the method of generating MPOV video may include, but is not limited to, the following steps: obtaining a plurality of media content; based on each meta-data corresponding to each of the media content And identifying, from the plurality of media content, the first media content and the second media content as related media content related to the same event, wherein the metadata includes at least time information or location information; and according to the related media content The MPOV video is generated.

In accordance with one of the exemplary embodiments, the present disclosure provides an electronic device that will include, but is not limited to, a processor configured to: obtain a plurality of media content; based on corresponding to the media content Identifying the first media content and the second media content from the plurality of media content as each piece of media content, as related media content related to the same event, wherein the metadata includes at least time information or location information And generating the MPOV video based on the related media content.

However, it should be understood that the summary is not intended to be exhaustive or to be construed as limiting. Moreover, the disclosure Improvements and modifications obvious to those skilled in the art will be included.

The above described features and advantages of the invention will be apparent from the following description.

10‧‧‧First electronic device

11‧‧‧First viewpoint

12‧‧‧ First media content

20‧‧‧Second electronic device

21‧‧‧second viewpoint

22‧‧‧Second media content

30‧‧‧ Third electronic device

31‧‧‧ Third viewpoint

32‧‧‧ Third Media Content

40‧‧‧Batterers

50‧‧‧Pitcher

100‧‧‧Electronic devices

110‧‧‧ processor

130‧‧‧Display screen

150‧‧‧Storage media

170‧‧‧Image capture component

190‧‧‧ transceiver

410‧‧‧Audio waveform

420‧‧‧ distinguishable features

421‧‧ ‧ time stamp

422‧‧‧ value

430‧‧‧ distinguishable features

440‧‧‧ distinguishable features

510‧‧‧AP1

520‧‧‧AP2

530‧‧‧AP3

540‧‧‧AP4

550‧‧‧AP5

560‧‧‧ first list

570‧‧‧Second list

610‧‧‧First media content

620‧‧‧Second media content

640‧‧‧ overlap

650 ‧ ‧ focus period

710‧‧‧First media content

720‧‧‧Second media content

721‧‧ frames

740‧‧‧ overlap

750‧‧ ‧ focus period

810‧‧‧First media content

820‧‧‧Second media content

850 ‧ ‧ focus period

910‧‧‧First media content

920‧‧‧Second media content

930‧‧‧ Third Media Content

940‧‧‧ overlap

950 ‧ ‧ focus period

S1010, S1011, S1012, S1013, S1014, S1020, S1030, S1110, S1120, S1130, S1140, S1150‧‧

T1‧‧‧ first time

T2‧‧‧ second time

T3‧‧‧ third time

T4‧‧‧ fourth time

T5‧‧‧ fifth time

1 illustrates a conceptual diagram of cooperative capture of events from different viewpoints to generate MPOV video in accordance with an exemplary embodiment of the present disclosure.

2A through 2D are diagrams illustrating that one of the embodiments according to the present disclosure is based on the first media content captured by the first electronic device 10, the second media content captured by the second electronic device 20, and the third electronic device. The third media content retrieved by device 30 produces a conceptual map of MPOV video.

3 is a block diagram illustrating the hardware of an electronic device in a functional block in accordance with one of the exemplary embodiments of the present disclosure.

4 is a diagram illustrating audio waveforms of media content in accordance with one of the exemplary embodiments of the present disclosure.

5A and 5B are diagrams illustrating concepts of ordering of signal strengths of nearby devices in accordance with one of the embodiments of the present disclosure.

6 is a diagram illustrating synchronization of first media content and second media content on a timeline in accordance with one of the exemplary embodiments of the present disclosure.

7 is a diagram illustrating synchronization of first media content and second media content when the first media content is a still image, according to one of the exemplary embodiments of the present disclosure. formula.

FIG. 8 is a diagram illustrating synchronization of first media content, second media content, and third media content in accordance with one of the exemplary embodiments of the present disclosure.

9 is a flow chart illustrating a method of generating MPOV video in accordance with one of the embodiments of the present disclosure.

10A and 10B are flow diagrams illustrating a method of generating MPOV video based on related media content, in accordance with one of the embodiments of the present disclosure.

11 is a flow diagram illustrating a method of generating MPOV video based on media content within a focused period, in accordance with one of the embodiments of the present disclosure.

Reference will now be made in detail to the present embodiments of the invention, Wherever possible, the same element symbols are used in the drawings and the description

By using the image capture functionality of electronic devices such as smart phones, tablets, wearable devices, etc., people can capture daily life events and store them as media content (eg, photos, video, audio). Etc.) Record these events. For example, people can use a smart phone, tablet, wearable device, camera, etc. to record a child in a baseball game. Therefore, media content related to the same baseball game can be captured by different users from different perspectives. The present disclosure provides a method of generating multi-viewpoint (MPOV) video that identifies relevance of media content, eg, how media content relates to events in time and location turn off. The relevant media content of the same event will then be used to generate MPOV video. In MPOV video, related media content retrieved from different viewpoints will be synchronized in time so that events taken from different viewpoints at almost the same time can be simultaneously presented in each frame of the MPOV video. The relevant media content can be further analyzed to identify a highlight period of the event such that the MPOV video will be generated based on relevant media content within the focused period.

1 illustrates a conceptual diagram of cooperative capture of events from different viewpoints to generate MPOV video in accordance with an exemplary embodiment of the present disclosure. Referring to Figure 1, a plurality of media content of an event (e.g., a shot event as shown in Figure 1) can be retrieved by a different electronic device. The media content will include the first media content retrieved from the viewpoint 11 by the first electronic device 10, the second media content retrieved by the second electronic device 20 from the viewpoint 21, and the third electronic device 30 retrieved from the viewpoint 31. Third media content.

2A through 2D are diagrams illustrating that one of the embodiments according to the present disclosure is based on the first media content captured by the first electronic device 10, the second media content captured by the second electronic device 20, and the third electronic device. The third media content retrieved by device 30 produces a conceptual map of MPOV video. Referring to FIGS. 1 and 2A, a batting event is retrieved from the first viewpoint 11 by the first electronic device 10, wherein the first media content 12 retrieved from the first viewpoint 11 will take a batting event from the side of the striking hand 40. Focus. Referring to FIGS. 1 and 2B, a batting event is captured by the second electronic device 20 from the second viewpoint 21, wherein the second media content 22 retrieved from the second viewpoint 21 will take a batting event from the back of the striking hand 40 as Focus. Referring to FIGS. 1 and 2C, a batting event is captured from the third viewpoint 31 by the third electronic device 30, wherein the third media content captured from the third viewpoint 31 is captured. 32 will focus on the pitcher 50 who throws the ball for the batting event.

Based on the first media content 12, the second media content 22, and the third media content 32 illustrated in FIG. 2A to FIG. 2C, the disclosure will identify the first media content 12 and the second media captured by different electronic devices from different viewpoints. Whether content 22 and third media content 32 will be associated with the same event in time and location. Assuming that the first media content 12, the second media content 22, and the third media content 32 are associated with the same event, then the first media content 12, the second media content 22, and the third media content 32 will be included to produce simultaneous presentations from different viewpoints. The MPOV video of the batting event is illustrated in Figure 2D. Referring to FIG. 2D, the MPOV video is simultaneously displayed from the first viewpoint 11, the second viewpoint 21, and the third viewpoint 31, and the MPOV will be combined/combined in the frame of the MPOV video. A media content 12, a second media content 22, and a third media content 32 display events in a collage style. The media content that displays the combination of the same event in a collage style will later be referred to as the collage view of the event. The frame of the MPOV video contains at least two parts, and each part can be used to display one of the media contents.

In one of the embodiments of the present disclosure, the frame of the MPOV video may be, but is not limited to, divided into three parts, namely, a left part, an upper right part, and a lower right part, and each part may be used to display by different electronic devices. Captured media content. For example, in the exemplary embodiment illustrated in FIG. 2D, the first media content 12 captured from the first viewpoint 11 is tiled to the left portion of the frame of the MPOV video, and the second portion captured from the second viewpoint 21 The media content 22 is tiled to the upper right portion of the frame of the MPOV video, and the third media content 32 captured from the third view 31 is tiled to the frame of the MPOV video. The lower right part. It should be noted that the media content may be a video or a photo, and the first media content, the second media content, and the third media content captured by the different electronic devices will be synchronized in time so that the events can be played at almost the same time.

The exemplary embodiments are for illustrative purposes and it is not desirable to limit the location or style of the collage view of the MPOV video. In another exemplary embodiment, media content of different viewpoints may be displayed equally in a frame of MPOV video or any other partition. The display position of each of the media content of different viewpoints can be arbitrarily positioned in the collage view of the event.

3 is a block diagram illustrating the hardware of an electronic device in a functional block in accordance with one of the exemplary embodiments of the present disclosure. Exemplary electronic device 100 can be a smart phone, a mobile phone, a digital camera, a tablet, a wearable device, or the like. Exemplary electronic device 100 can include at least, but not limited to, processor 110, display screen 130, storage medium 150, image capture component 170, and transceiver 190. Each element of the exemplary electronic device 100 is explained in detail below.

The processor 110 can be, but is not limited to, a central processing unit (CPU) for general purpose or special purpose, or a programmable microprocessor, a digital signal processor (DSP), programmable A controller, an application specific integrated circuit (ASIC), a programmable logic device (PLD), or the like, or a combination thereof. In the present embodiment, the processor 110 can be electrically coupled to the display screen 130, the storage medium 150, the image capture component 170, and the transceiver 190, respectively, wherein the processor 110 will control all operations of the exemplary electronic device 100.

The display screen 130 may be a display device that provides a display function within a display area of the electronic device 100. The display device can be, but not limited to, a liquid crystal display (LCD), a light-emitting diode (LED), a field emission display (FED), or the like.

The storage medium 150 can be a volatile or non-volatile memory for storing buffered data or permanent data, such as media content retrieved by the image capture component 170 or for performing exemplary mobile electronic devices. 100 function instructions.

The image capturing component 170 can be, but is not limited to, a camera, a video camera, etc., which captures scenes as media content (eg, photos, videos, etc.) through optical components and imaging components. In the present disclosure, media content representing a scene of related events may be captured by image capture component 170 and stored in storage medium 150.

The transceiver 190 can support a global system for mobile communication (GSM), a user's personal handy-phone system (PHS), a code division multiple access (CDMA) system, Wideband code division multiple access (WCDMA) system, long term evolution (LTE) system, worldwide interoperability for microwave access (WiMAX) system, wireless fidelity (wireless fidelity, Wi-Fi) System or component of Bluetooth signal transmission, for example, a protocol unit. Transceiver 190 can also be a component for its support. The transceiver 190 will provide wireless transmission for the electronic device 100, including multiple components, but is not limited to transmitter circuitry, receiver circuitry, analog/digital (A/D) converters, digital/analog ratio (D/A) converters, Low noise amplifier Amplifier, LNA), mixer, filter, matching network, transmission line, power amplifier (PA) and one or more antenna elements. The transmitter and receiver wirelessly transmit downlink signals and receive uplink signals. The receiver may include functional elements that perform operations such as low noise amplification, impedance matching, mixing, upconversion, filtering, power amplification, and the like. Analog/digital (A/D) or digital/analog (D/A) converters are configured to convert from analog signal format to digital signal form during uplink signal processing and from digital signal format during downlink signal processing Convert to analog signal format. In an exemplary embodiment of the present disclosure, the transceiver 190 can be used to transmit media content to or receive media content from different electronic devices.

In the following, MPOV video will be generated in detail with respect to the exemplary media content illustrated in FIG. 1 and FIGS. 2A through 2D with respect to the exemplary electronic device 100 illustrated in FIG.

Referring to FIG. 3, the scene of the event can be captured by the image capture component 170 of the exemplary electronic device 100, and then the processor 110 views the scene of the event as multiple media content (eg, images, continuous images, audio). Audio recording, etc., is stored in the storage medium 150 of the exemplary electronic device 100. It should be noted that a continuous image may refer to a plurality of image frames in a video or a plurality of images captured in a continuous shooting mode.

The exemplary electronic device 100 can use the processor 110 to identify the relevance of the media content and then generate MPOV video based on the media content associated with the event. In one of the exemplary embodiments of the present disclosure, the first media content and the second media content in the media content are used as examples for illustration. The processor 110 of the electronic device 100 will use the first media content and the second media based on time information and/or location information. Body content is identified as being related to the same event. In detail, the processor 110 extracts time information and/or location information from metadata respectively embedded in the first media content and the second media content or associated with the first media content and the second media content, respectively. And determining whether the first media content is related to the second media content in time and/or location. It should be noted that this embodiment of the present disclosure does not limit the source of media content. That is, the media content can include media content retrieved by the exemplary electronic device 100 or captured by other nearby electronic devices (eg, any of the electronic devices 10, 20, 30 illustrated in FIG. 1). Media content transmitted.

Moreover, in one of the exemplary embodiments of the present disclosure, the audio information of the first media content and the second media content can be used to identify whether the first media content and the second media content are identical in time and/or location Event related.

In one of the exemplary embodiments, time information (eg, timestamp), audio information, and location information (with geotags and surrounding signal information) may be obtained, the information corresponding to the media content being captured Everyone in the media content. In the following, time information, audio information, and location information will be described in detail.

The time information of the media content may include, but is not limited to, a time stamp that records the date and time of the moment of capturing each of the media content. The time stamp can be obtained from the system clock of the electronic device 100, wherein the system clock can be a Global Positioning System (GPS), a Wi-Fi access point, a radio access network, and a servo. The device is automatically synchronized. However, the exemplary embodiments are not intended to limit the disclosure, and the time stamps may be configured by the user or by any other means.

The audio information may include, but is not limited to, information about the audio waveform of the media content, such as a waveform shape, a distinguishing feature having a value at a particular time, and the like. 4 is a diagram illustrating an audio waveform 410 of media content in accordance with one of the exemplary embodiments of the present disclosure. Referring to FIG. 4, distinguishable features 420, 430, 440 are available from audio waveform 410. In one of the exemplary embodiments, the value 422 and the timestamp 421 corresponding to the value 422 may be obtained from the distinguishable feature 420 and then stored after the capture of each of the media content corresponding to the audio information. In the metadata of the media content. However, the type of audio information embedded in the metadata is not limited to that shown in the above exemplary embodiment, and other information related to the audio waveform of the media content may be utilized. In one of the exemplary embodiments, the shape of the background noise can also be utilized. Moreover, the disclosure is not limited to the exemplary embodiments described. In one of the exemplary embodiments, audio information may be extracted from the media content, and the audio information simultaneously identifies the relevance of the media content. That is, instead of embedding audio information into the metadata of the media content, the processor 110 can analyze the media content in real time to extract audio information of the media content to identify whether the media content is associated with the same event.

Location information can include, but is not limited to, geotags and surrounding signal information. Geotags can include, but are not limited to, GPS location, accuracy data, and the like. The GPS position and accuracy data can be obtained from a GPS chip (not shown) of the electronic device that retrieves the media content after the media content is captured. The GPS location of the media content may record, but is not limited to, a longitude coordinate and a latitude coordinate representing the location at which the media content was retrieved. The accuracy data will record the accuracy of the longitude coordinates and latitude coordinates when acquiring the corresponding GPS position.

The surrounding signal information of the location information may include, but is not limited to, related devices (eg, other mobile electronic devices (eg, smart phones in hotspot mode), access points (APs, eg, Wi-Fi routers), radios Information on the signal strength of a network access tower, etc.). In other words, a wireless signal (eg, Wi-Fi, Bluetooth, or radio signal) between the electronic device 100 and a nearby device can be used to determine the relative distance between the locations of each of the captured media content. In one of the exemplary embodiments, there are a plurality of wireless devices around the electronic device 100 that captures the media content, and the signal strength of each of the nearby devices of the electronic device 100 can be analyzed and the signal strength is performed. Sort to form a list that sorts the signal strength of nearby devices. For example, when each of the media content is retrieved, a list of the signal strengths of nearby devices can be embedded in the metadata of each of the media content.

5A and 5B are diagrams illustrating concepts of establishing a list of nearby devices ranked based on signal strength, in accordance with one of the embodiments of the present disclosure. 5A and 5B, it is assumed that the access point (AP) 1510, AP2 520, AP3 530, AP4 540, and AP5 550 are the nearby first electronic device 10 and second electronic device 20. In the described embodiment, the first electronic device 10 and the second electronic device 20 will sort nearby devices based on signal strength after the media content is captured. For example, a first list 560 can be generated, which sorts the signal strengths of each of the nearby devices with respect to the first electronic device 10 in a sequence of AP1 510, AP3 530, AP2 520, AP5 550, etc., and A list 560 can be embedded in the metadata of the media content retrieved by the first electronic device 10. A second list 570 can also be generated, which is AP1 510, AP5 550, The sequence of AP3 530, AP4 540, etc. sorts the signal strength of each of the nearby devices with respect to the second electronic device 20, and the second list 570 can be embedded in the element of the media content retrieved by the second electronic device 20. In the information. In this embodiment, the basic service set identification (BSSID) of each of the APs is used to identify the AP in a list of ranked signal strengths containing nearby devices. However, the disclosure is not limited thereto, because the AP can be identified by other means.

Moreover, in an alternative embodiment of the present disclosure, the signal strength between the first electronic device 10 and the second electronic device 20 can also be used to determine whether the first media content 12 captured by the first electronic device 10 is in positional The second media content 22 retrieved by the second electronic device 20 is associated.

In the following, the correlation between the first media content and the second media content is identified based on time information, audio information, and location information.

In this embodiment, the processor 110 will identify whether the first media content and the second media content are temporally related to the same event based on the time code embedded in the metadata. In detail, the processor 110 obtains a time code indicating the time stamp of the time of capturing the first media content and the second media content from the metadata, and determines whether the first media content and the second media content are related to the same event. . In one of the exemplary embodiments of the present disclosure, the processor 110 may calculate a time difference between a time code of the first media content and the second media content, and determine between the first media content and the second media content. Whether the time difference is within the predetermined range. For example, the predetermined range can be 2 hours. If the time difference between the first media content and the second media content is within 2 hours, the processor 110 will consider the first media content and the second media content to be temporally related Related to the same event.

However, the disclosure is not limited thereto. In one of the exemplary embodiments of the present disclosure, when the time code of the first media content and the second media content is within a predetermined time period indicating the occurrence of the event, the first media content and the second media content are to be identified For correlation in time. For example, a baseball event may occur between 4:00 pm and 9:00 pm on May 1, and the predetermined time period may be configured, for example, from 4:00 pm to 9:00 pm. If the time code indicates that the first media content and the second media content are captured between 4:00 pm and 9:00 pm, the first media content and the second media content will be identified as being related to the baseball event in time. . It should be noted that the predetermined time period may be automatically determined by the media content or configured by the user. For example, there may be periods in which media content is not captured before a baseball event and after a baseball event. The processor 110 will automatically detect the time interval between these periods of uncaptured media content as a predetermined time period, and group the media content captured during the predetermined time period into a media mix. It should be noted that the predetermined time period will conveniently serve as one of the parameters for identifying whether any media content is temporally related to a set of relevance criteria.

In the present disclosure, the processor 110 will also identify whether the first media content and the second media content are related in position to the same event based on the audio waveform 410, the geotag, or the surrounding signal information. A detailed description of identifying the location relevance of the media content is described below.

To determine if the media content is locationally related to the event, the processor 110 can utilize the distinguishable features 420, 430, 440 of the audio waveform 410 of the first media content and the second media content, which are illustrated in FIG. For example, in the batting event, The distinguishing feature 410 can be the noise generated when the baseball contacts the bat (ie, the hitting noise), and the distinguishing features 420, 430 can be cheering noise generated by the crowd. The processor 110 correlates the first media content and the second media content that are considered to have hitting noise and cheering noise in position with the same event.

In one of the exemplary embodiments of the present disclosure, the first media content and the second media content may be identified as being related in position to the same event based on the geographic tag. The processor 110 may obtain geotags, such as GPS location and accuracy data, from metadata of the first media content and the second media content. The processor 110 will identify the first media content and the second media content as being related in position to the same event by utilizing the GPS location of the first media content and the second media content. For example, processor 110 will determine if the difference between the GPS locations of the first media content and the second media content is within a predetermined distance. If the difference is within a predetermined distance, then the first media content and the second media content are considered to be related in position to the same event. The predetermined distance may be configured according to the actual application; the disclosure does not wish to limit the range of the predetermined distance, wherein the range may be configured for any value within 500 meters for the baseball field or any value within 5 kilometers for the track. .

In one of the exemplary embodiments of the present disclosure, the processor 110 may identify the first media content and the second media content based on the GPS location of the first media content and the second media content and a predetermined geographic coordinate of the related event. Related to the same event. The predetermined geographic coordinates may be GPS coordinates indicating where an event has occurred, which may be automatically obtained from GPS locations that have been identified as media content associated with the event. The disclosure does not wish to limit the acquisition of predetermined geographic coordinates of related events, indicating where an event has occurred The GPS coordinates can also be manually configured by the user. For example, a user can manually enter a GPS coordinate of a particular location (eg, a baseball field) as a predetermined geographic coordinate. In an exemplary embodiment, processor 110 determines whether the GPS location of the first media content and the second media content is within a predetermined distance with respect to a predetermined geographic coordinate of the relevant event. In the described embodiment, the predetermined geographic coordinates may be one of the parameters of the set of correlation criteria.

In addition, processor 110 will analyze the accuracy data of the geotag to determine if the accuracy of the GPS location is within a predetermined range. In other words, the processor 110 will determine if the GPS location obtained when capturing the first media content and the second media content is trustworthy. In the present exemplary embodiment, the predetermined range may be configured as, but not limited to, any range of values within 100 meters. That is, when the accuracy information of the geotag is within a predetermined range, the processor 110 uses the GPS location to identify whether the first media content and the second media content are related in position to the same event. On the other hand, if the accuracy data indicates that the GPS location is not within the predetermined range, the processor 110 will identify whether the first media content and the second media content are related to the event, regardless of the GPS location of the geotag.

In one of the exemplary embodiments, processor 110 will identify whether the first media content and the second media content are related to the same event based on the surrounding signal information. In detail, the processor 110 obtains a list 560, 570 that sorts nearby devices based on signal strength from the metadata of the first media content and the second media content. In the embodiment illustrated in Figures 5A and 5B, the lists 560, 570 will contain nearby devices sorted from high to low based on the signal strength of nearby devices to the electronic device. Set. The exemplary embodiment includes first electronic device 10 and second electronic device 20 that take an event of interest as a plurality of media content from different viewpoints.

For example, the first electronic device 10 will capture the first media content, and the second electronic device 20 will capture the second media content. In the vicinity, there are AP1 510, AP2 520, AP3 530, AP4 540, and AP5 550. Processor 110 may determine whether the first media content and the second media content are locationally related to the same event based on the ranking of the APs listed in lists 560 and 570. The list 560 will sort the APs around the first electronic device 10 after the first media content is captured based on the signal strength between the AP and the first electronic device 10, and the list 570 will be based on the AP and the second electronic device 20 The signal strength between the two is sorted by the AP around the second electronic device 20 after the second media content is captured. In an exemplary embodiment, processor 110 will compare the ordering of AP1 510, AP2 520, AP3 530, AP4 540, and AP5 550 between lists 560 and 570.

For example, in one of the exemplary embodiments, at least three of the APs listed in lists 560 and 570 must be the same, and at least two of the at least three APs must be The top three in the list are such that the processor 110 considers the first media content and the second media content to be positionally related to the same event based on the signal strength of nearby devices. However, the present disclosure does not limit requirements for identifying the relevance of the first media content and the second media content based on surrounding signal information, which may be designed according to the actual application. For example, in other exemplary embodiments, the requirements for identifying the first media content and the second media content related to the same event may be two of the matching APs in the list and any of the two matching APs Are listed in the list The first three.

In one of the exemplary embodiments of the present disclosure, the relevance of the media content may also be identified from the Wi-Fi Direct signal in the absence of an AP in the vicinity. For example, the exemplary electronic device 100 can have the ability to communicate with other nearby electronic devices via Wi-Fi Direct in a hotspot mode. The Wi-Fi Direct signal strength between the exemplary electronic device 100 and other nearby electronic devices can be detected and stored in the metadata of each of the media content. Then, when it is identified that the first media content and the second media content are related in position to the same event, the processor 110 of the exemplary electronic device 100 can utilize the Wi-Fi Direct signal strength of other nearby electronic devices to determine that the difference is different. Whether the media content captured by the electronic device is related to each other.

Based on the above, the present disclosure will identify that the first media content and the second media content are related to the same event in time and location according to the time code, the audio waveform of the nearby device, the geotag, and the signal strength, and the detailed description of each step Above. In one of the embodiments of the present disclosure, the processor 110 may first identify whether the first media content and the second media content are related to the same event based on the time code. If the first media content and the second media content are not temporally related, the processor 110 will consider that the first media content and the second media content are also not related in the same location to the same event. If it is determined that the first media content and the second media content are temporally related, the processor 110 will then determine whether the first media content and the second media content are in a location with a sequence of audio waveforms, geotags, and signal strengths of nearby devices. Related to the same event. However, the disclosure is not limited thereto. Identify the first media content and the second media Whether the volume content is temporally and geographically related to the same event sequence can be modified to meet the design requirements of the actual application.

Once the first media content and the second media content are identified as being associated with the same event, the processor 110 will then provide the first media content and the second media content as candidate media content for generating MPOV video.

After identifying the relevance of the first media content and the second media content, the exemplary electronic device 100 will further synchronize the first media content and the second media content based on the time code and the audio information. In detail, the processor 110 will adjust the first media content and the second media content according to the time stamp of the first media content and the time stamp of the second media content. For example, the first media content and the second media content may be adjusted according to a start time and an end time of each of the first media content and the second media content.

In addition, the processor 110 can also adjust the first media content and the second media content according to the audio waveform. As described above, information about the audio waveform of the media content can be extracted from the media content, such as the value of the distinguishable feature of the audio waveform and the timestamp corresponding to the value. In the example of the above-described batting event, the first media content and the second media content may be adjusted based on the shot noise generated when the bat contacts the ball. For example, processor 110 will identify a value representative of the hitting noise in the first media content and the second media content (eg, the value 422 illustrated in FIG. 4), and then based on a time stamp corresponding to the value ( For example, the time stamp 421 illustrated in FIG. 4 aligns the first media content and the second media content on a unified time axis. However, the disclosure is not limited thereto. The first medium can be aligned according to other characteristics of the audio waveform (eg, the shape of the distinguishable feature) Body content and second media content.

Once the first media content and the second media content are synchronized in time, the processor 110 will identify overlapping portions of the first media content and the second media content that overlap in time. In one of the exemplary embodiments, the overlapping portion may be identified based on the start time and end time of the first media content and the second media content.

Thereafter, the processor 110 will further identify the focus period associated with the overlapping portion based on the content characteristics of the first media content or the second media content. The content feature may refer to a media type and a lens type of the media content, wherein the media type of the media content may be a still image or a continuous image, and the lens type of the media content may be a closed-up shot or a medium shot ( Medium shot), zoom-in shot, or zoom-out shot. The media type and lens type of the media content can be used to automatically determine the focus of the event.

In one of the exemplary embodiments, face detection may be implemented to determine the ratio between the portion of the frame occupied by the object and the other portions of the frame that are not occupied by the object. If the ratio of objects occupying the frame exceeds a predetermined ratio, the processor 110 identifies the shot type of the frame of the second media content as a close-up shot. For example, the predetermined ratio can be any value, for example, 60% or more. When the object occupies at least 60% of the frame, the processor 110 identifies the frame with the object occupying at least 60% of the frame as a close-up. In addition, processor 110 will also identify mid-range shots (eg, empty shots) for frames having objects that occupy less than at least 60% of the area. In the case of continuous images, the above technique can be used to determine whether the lens type of the video is a near lens or a far lens by analyzing an area occupied by the object in each frame of the video.

In one of the exemplary embodiments, the media type of the media content is used to identify a focus period associated with the overlapping portion. For example, a scene that is an event captured by a still image associated with an overlapping portion will be considered the focus of the event. In one of the embodiments of the present disclosure, the lens type of the media content can be used to identify a focus period associated with the overlapping portion. The focus period can be identified based on the media content captured in the close-up or close-up type because the user can tend to take close-up or close-up shots of the moments that are considered to be the focus of the event.

In the following, FIGS. 6 through 8 are specific exemplary embodiments for illustrating details of highlighting periods of emphasis within overlapping portions of first media content and second media content.

6 is a diagram illustrating synchronization of first media content and second media content on a unified timeline in accordance with one of the exemplary embodiments of the present disclosure. When the first media content 610 and the second media content 620 are synchronized in time, the processor 110 will further identify the overlapping portion 640, wherein the first media content 610 and the second media content 620 are at the first timestamp T1 and the second time The stamps T2 overlap in time. In other words, the overlapping portion between the first media content 610 and the second media content 620 will represent the same real life event that was drawn from a different perspective at substantially the same time. In this embodiment, the overlapping portion of the first media content 610 and the second media content 620 may be identified according to the start time of the second media content 620 and the end time of the first media content 610.

Referring to FIG. 6, the first media content 610 and the second media content 620 are continuous images having overlapping portions 640 between the first time T1 and the second time T2. even The continuation image will contain multiple frames. The processor 110 will analyze the first media content 610 and the second media content 620 within the overlap portion 640 to determine the media type and shot type to identify the highlight period associated with the overlap portion 640. In detail, the processor 110 will determine that the first media content 610 and the second media content 620 are continuous images. Moreover, the processor 110 will identify the second media content 620 as a close-up of the object because the object captured by the second media content 620 (eg, the striking hand) occupies a larger area than the second associated with the overlapping portion 640 A predetermined ratio of frames of media content 640 (eg, 60%). The area occupied by the object is similar throughout each frame of the second media content 620 associated with the overlapping portion 640, and thus the lens type of the second media content 620 is determined to be a close-up.

In the exemplary embodiment illustrated in FIG. 6, processor 110 will identify focus period 650 within overlap portion 640 because the media type of second media content 620 is identified as a close-up shot. That is, the area occupied by the object within the frame of the second media content 620 associated with the overlapping portion is greater than a predetermined ratio. It should be noted that the overlapping portion 640 can have a continuous footage of events of greater length (eg, 5 minutes). The processor 110 can define the focus period 650 for a predetermined duration (eg, any duration less than 5 minutes). In the present exemplary embodiment, the focus period 650 may be defined between the third time T3 and the fourth time T4. The highlight period 650 defined by the third time T3 and the fourth time T4 may be arbitrarily selected within the overlap portion 640 and may have a predetermined duration. The present disclosure does not wish to limit the duration and selection of focus periods 650 within overlapping portions 640 of media content having close-up types.

FIG. 7 is a diagram illustrating the unification of one of the exemplary embodiments in accordance with the present disclosure. A synchronized schema of the first media content 710 and the second media content 720 on the timeline. In an exemplary embodiment, the second media content 720 is a continuous image captured in a near shot type and a far shot type, wherein the area occupied by the object in the frame of the second content 720 increases and decreases. As noted above, the user may tend to zoom in at the object at a time that is considered the focus of the event. Accordingly, processor 110 will analyze second media content 720 and identify frame 721 of second media content 720 within overlapping portion 740 as a focus. Next, the processor 110 identifies the time period having the predetermined duration of the time stamp defined by the third time T3 and the fourth time T4 and centered on the frame 721 of the second media content as the focus period 750, because the object is at The area occupied by the frame 721 of the second media content 720 exceeds a predetermined ratio (for example, 60%). Accordingly, processor 110 will select first media content 710 and second media content 720 within highlight period 750 as candidate sources for generating MPOV video.

FIG. 8 is a diagram illustrating synchronization of first media content 810 and second media content 820 when first media content 810 is a still image, in accordance with one of the exemplary embodiments of the present disclosure. In an exemplary embodiment, the first media content 810 is a still image and the second media content 820 is a continuous image (eg, a video or continuous image). The processor 110 will identify the overlapping portion, wherein the first media content 810 overlaps the second media content 820 at a fifth time T5 defined by the timestamp of the first media content 810. The processor 110 will identify a focus period 850 that is centered on the timestamp of the first media content 810 between the third time T3 and the fourth time T4 because the media type of the first media content 810 is a still image. In other words, the processor 110 may select a time interval having a predetermined duration before and after the fifth time T5 as a focus. Period 850. However, the disclosure is not limited thereto, and the key period 850 can be selected by other means. Accordingly, processor 110 will select first media content 810 and second media content 820 within highlight period 850 as candidate sources for generating MPOV video.

9 is a diagram illustrating synchronization of first media content 910, second media content 920, and third media content 930 in accordance with one of the exemplary embodiments of the present disclosure. In an exemplary embodiment, the second media content 920 as a continuous image and the third media content 930 as a still image are captured by the same electronic device during the same event period. The third media content 930 can be obtained while the second media content 920 is being captured. The processor 110 will identify the overlapping portion 940, wherein the first media content 910 overlaps the second media content 920 and the third media content 930 between the first time T1 and the second time T2. Processor 110 will then identify a focus period 950 that is defined between the third time T3 and the fourth time T4 and is centered on the timestamp of the third media content 930. The determination of the time interval of the focus period 950 is similar to the exemplary embodiment illustrated in FIG. 8, and thus is omitted herein. Accordingly, processor 110 will select first media content 910, second media content 920, and third media content 930 within highlight period 950 as candidate sources for generating MPOV video.

10A and 10B are flow diagrams illustrating a method of generating MPOV video in accordance with one of the embodiments of the present disclosure. Referring to FIG. 10A, in step 1010, the exemplary electronic device 100 will obtain a plurality of media content. In step S1020, the exemplary electronic device 100 will identify the first media content and the second media content from the plurality of media content based on each meta-data corresponding to each of the media content, as Relevant media content related to the same event, wherein the meta-data includes at least time information or location information. In step S030, the exemplary electronic device 100 will generate multi-view (MPOV) video based on the relevant media content.

Details regarding the identification of the first media content and the second media content of step S1020 in FIG. 10A will be described below in FIG. 10B. Referring to FIG. 10B, in step S1011, the exemplary electronic device 100 will identify the first media content and the second media content as being temporally identical by comparing the time code of the first media content with the time code of the second media content. Relevant media content related to the event. In step S1012, the exemplary electronic device 100 may identify the first media content and the second media content as related media content related in position to the same event by comparing the audio information of the first media content and the second media content. In step S1013, the exemplary electronic device 100 may determine whether the accuracy data is within a predetermined accuracy range, and then determine the GPS position of the first media content and the GPS position of the second media content when the accuracy data is within the predetermined accuracy range. Whether the difference is within a predetermined distance, and the first media content and the second media content are identified as related media content that is related in position to the same event. In step S1014, the exemplary electronic device 100 may identify the first media content and the second media content as being in position by comparing the order of the nearby devices between the list of the first media content and the list of the second media content. Relevant media content related to the same event.

11 is a flow diagram illustrating a method of generating MPOV video based on media content within a focused period, in accordance with one of the embodiments of the present disclosure. In the exemplary embodiment illustrated in FIG. 11, steps S1110 and S1120 will be similar to that illustrated in FIG. Steps S1010 and S1020, and therefore, the description of steps S1110 and S1120 are omitted.

Referring to FIG. 11, in step S1130, the exemplary electronic device 100 will identify overlapping portions of the first media content and the second media content that overlap in time. In step S1140, the exemplary electronic device 100 will identify the focus period by analyzing the content characteristics of the first media content and the second media content associated with the overlapping portion. In step S1150, the exemplary electronic device 100 generates MPOV video from the first media content and the second media content in the focus period.

In the following, the exemplary electronic device 100 will be further elaborated with specific exemplary embodiments with reference to the methods illustrated in FIGS. 10A-10B and 11. The present embodiment of the present disclosure will determine, based on the steps illustrated in Figures 10A and 10B, whether each of the media content is related to an event in time and location, and is identified as being associated with the same event in time and location. Media content will be identified as relevant media content. Next, the exemplary electronic device 100 will provide media content that is identified as being associated with the same event as candidate media content for generating MPOV video. Moreover, in accordance with the steps illustrated in FIG. 11, exemplary electronic device 100 will synchronize relevant media content to identify overlapping portions of temporally overlapping related media content. Next, the highlight period will be identified within the overlap to generate MPOV video with the focus of the event.

In one of the exemplary embodiments of the present disclosure, the exemplary electronic device 100 illustrated in FIG. 3 can be any of the electronic devices 10, 20, 30 illustrated in FIG. In other words, the media content representing the scene of the event may be captured by the image capture component 170 of each of the electronic devices 10, 20, 30 from different viewpoints of the event, And stored in the storage medium 150 of each of the electronic devices 10, 20, 30.

Referring to FIG. 1, the first electronic device 10 may be a content requester that initiates a process of generating MPOV video and requests media content, and the second electronic device 20 and/or the third electronic device 30 may provide media content to the electronic device. 10 to the content provider of the MPOV video that generated the event. In an exemplary embodiment, the first electronic device 10 may respond to time code, audio information, and location information (including geotags and surrounding signal information) embedded in the metadata of the media content or associated with the metadata of the media content. And identifying related media content in multiple media content. In one of the exemplary embodiments of the present disclosure, the first electronic device 10 may group the related media content into a media mix. It should be noted that grouping media content into media mixes (eg, media sets) may be performed automatically based on metadata after the media content has been retrieved or manually by the user, for example, inserting the media content into the compilation or removing the media content. Execution, the disclosure is not limited to this.

Next, the first electronic device 10 may transmit the metadata of the relevant media content as a set of relevance criteria to the second electronic device 20 and/or the third electronic device 30 to request relevant media content (ie, metadata exchange). In this embodiment, the set of correlation criteria may include time information, audio information, and location information. The second electronic device 20 will identify the relevant media content among the plurality of media content retrieved by the second electronic device 20 based on the metadata of the related media content transmitted from the first electronic device 10. In other words, the second electronic device 20 will recognize the media content captured by the second electronic device 20 in response to the time information, the video information, and the location information corresponding to the related media content captured by the first electronic device 10. Relevant media media. In addition, the third The electronic device 30 will execute a program similar to the electronic device 20, and therefore, a description about the third electronic device will be omitted.

It should be noted that the disclosure is not limited to the above meta-data exchange. In one of the exemplary embodiments, the first electronic device 10 may obtain a set of correlation criteria based on media content within the media mix. In another exemplary embodiment, the set of correlation criteria may be configured by a user to indicate an interest of the user. For example, a media mix can be created to collect media content captured at a baseball stadium for a baseball game between 5:00 PM and 9:00 PM on May 1. Thus, a particular geographic coordinate for a predetermined time period (eg, 5:00 pm to 9:00 pm on May 1st) and a particular location of interest (eg, a baseball field) may be determined based on media content within the media mix. The audio information of the set of correlation criteria can be obtained by analyzing the relevant media content after the start of the generation of the MPOV video. For example, audio information of the media content within the media mix may be extracted as one of the parameters of the set of correlation criteria after the initiation of the generation of the MPOV video. Additionally, ambient signal information at a particular location may be obtained from metadata of media content within the media mix or manually configured by the user. The first electronic device 10 will then receive relevant media content corresponding to the set of correlation criteria from the second electronic device 20 or the third electronic device 30 as candidate (related) media content for the MPOV video.

Furthermore, the first electronic device 10 will synchronize the relevant media content to identify overlapping portions of the temporally overlapping related media content retrieved by the first electronic device 10, the second electronic device 20, and the third electronic device 30, and then based on the overlap The content characteristics of the media content within the portion identify the key period. Processor 110 will then pass through the combination / MPOV video is generated by mixing relevant media content during the key period.

In general, the exemplary electronic device of the present disclosure will associate time information, audio information, and location information in metadata associated with each of the media content or metadata associated with each of the media content. Identifying the first media content and the second media content related to the same event in time and location (including the geotag and surrounding signal information). Next, the first media content and the second media content will be provided as the relevant media content of the MPOV video used to generate the event. The present disclosure will further synchronize relevant media content to identify overlapping portions of related media content that overlap in time. From the overlapping portion, the key period of the event can be identified based on the content characteristics of the relevant media content. Therefore, MPOV video can be generated based on relevant media content during the key period.

The elements, acts or instructions in the detailed description of the disclosed embodiments of the present application should not be construed as being critical or essential to the present disclosure unless explicitly described. Moreover, as used herein, the word "a" can encompass more than one item. If you intend to refer to only one item, the term "single" or similar language will be used. Moreover, as used herein, the term "any of" preceding a list of items and/or plurality of item categories is intended to encompass the item and/or item type individually or in combination with other items and/or Any of the other item categories "any of the combinations", "any combination of", "any of the plurality", and/or any combination of the plurality. Also, as used herein, the term "set" is intended to encompass any number of items, including zero. Also, as used herein, the term "amount" is intended to include any quantity, including zero.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S1010~S1030‧‧‧Steps

Claims (20)

A method for generating multi-viewpoint (MPOV) video for an electronic device, the method comprising: obtaining a plurality of media content; and from the plurality of media based on each meta-data corresponding to each of the media content The content identifies the first media content and the second media content as related media content related to the same event, wherein the metadata includes at least time information or location information, wherein the first media content is based on a first viewpoint Taking the second media content according to a second viewpoint, wherein the first viewpoint is different from the second viewpoint; generating the multi-view video according to the related media content; Multi-view video, displaying the first media content and the second media content related to the same event by a display screen. The method of claim 1, further comprising: comparing the first media content and the second media content by comparing audio information between the first media content and the second media content Recognized as the related media content. The method of claim 2, wherein the audio information comprises a value and a time stamp of distinguishable features of the audio waveform. The method of claim 1, wherein determining the related media content further comprises: comparing a time code of the first media content with the second media content The first media content and the second media content are identified as time code to the related media content that is temporally related to the same event. The method of claim 1, wherein the location information comprises a geotag, and the geotag includes global positioning system location and accuracy data, and the first media content and the content are from the media content Determining, by the second media content, the related media content related to the same event includes: determining whether the precision data is within a predetermined accuracy range, and determining, by the determining, that the precision data is within the predetermined accuracy range Identifying whether the difference between the global positioning system location of the media content and the global positioning system location of the second media content is within a predetermined distance, and identifying the first media content and the second media content The related media content related to the same event in position. The method of claim 1, wherein the location information comprises surrounding signal information, and the surrounding signal information comprises a list of signal strengths of nearby devices, and the correlation is determined from the media content. The media content further includes: comparing the nearby devices listed in the list of the first media content and the nearby devices listed in the list of the second media content The first media content and the second media content are identified as the related media content that is related in position to the same event. The method of claim 6, wherein the nearby device comprises an access point or other electronic device. For example, the method described in claim 1 of the patent scope further includes: Transmitting a set of correlation criteria to another electronic device, the set of correlation criteria for identifying third media content retrieved by the another electronic device as related to the first media content and the One of the related media content of the second media content being the same event; and receiving the third media content from the another electronic device to generate the multi-view video. The method of claim 8, wherein the set of correlation criteria is determined based on the time information, audio information, and the location information of the first media content and the second media content. of. The method of claim 1, wherein the first media content and the second media content comprise still images, continuous images, and audio recordings. A video processing device, comprising: a processor configured to: obtain a plurality of media content; identify a first media content from the plurality of media content based on each meta-data corresponding to each of the media content a second media content, as related media content related to the same event, wherein the metadata includes at least time information or location information, wherein the first media content is captured according to a first viewpoint, the second media The content is captured according to a second viewpoint, wherein the first viewpoint is different from the second viewpoint; the multi-view video is generated according to the related media content; and according to the multi-view video, by one Display screen synchronization display is the same as the above The first media content related to the event and the second media content. The video processing device of claim 11, wherein the processor is further configured to compare the first media by comparing audio information between the first media content and the second media content The content and the second media content are identified as the related media content. The video processing device of claim 12, wherein the audio information comprises a value and a time stamp of distinguishable features of the audio waveform. The video processing device of claim 11, wherein the processor is further configured to compare the first time of the first media content with a time code of the second media content The media content and the second media content are identified as the related media content that is temporally related to the same event. The video processing device of claim 11, wherein the location information comprises a geotag, and the geotag includes GPS location and accuracy data, and the processor is further configured to determine whether the accuracy data is Determining a difference between the GPS location of the first media content and the GPS location of the second media content within a predetermined accuracy range and when the accuracy profile is within the predetermined accuracy range Whether the first media content and the second media content are identified as the related media content that is related in position to the same event, within a predetermined distance range. The video processing device of claim 11, wherein the location information includes surrounding signal information, and the surrounding signal information includes a list of signal strengths of nearby devices, and the processor is further configured. Through Comparing the first media content and the nearby devices listed in the list of the second media content and the nearby devices listed in the list of the second media content The second media content is identified as the related media content that is related in position to the same event. The video processing device of claim 16, wherein the nearby device comprises an access point and other electronic devices. The video processing device of claim 11, further comprising: a transceiver configured to: transmit a set of correlation criteria to another electronic device, the set of correlation criteria being used by the other Identifying, by the electronic device, the third media content as one of the related media content related to the same event as the first media content and the second media content; and from the another electronic device Receiving the third media content to generate the multi-view video. The video processing device of claim 18, wherein the set of correlation criteria is based on the time information, audio information, and the location information of the first media content and the second media content. To determine. The video processing device of claim 11, wherein the first media content and the second media content comprise a still image, a continuous image, and an audio recording.