TWI423051B - Semantic-based visual effect generating system and program product - Google Patents

Description

Semantic-based visual effects generation system and program products

The invention relates to a visual effect generating system, in particular to a semantic based visual effect generating system and a program product.

Due to the rapid development of the Internet, various multimedia materials can be obtained through the Internet platform; however, there are also some semantic analysis techniques for semantic analysis of such multimedia materials to obtain related semantic data (or interpret data ( Metadada)), the aforementioned semantic data is generally used to describe, classify, or organize the multimedia materials associated therewith to accelerate subsequent retrieval of the multimedia materials.

An automated image interpretation and retrieval system as disclosed in US Pat. No. 7,052,250, which analyzes digital images and video data, and interprets digital images and video data with a semantic label for subsequent use, for example, based on Content-based retrieval, and video abstract generation.

However, the above semantic data can reflect the context, emotion, image and other connotations of the multimedia materials in addition to the purpose of the retrieval; if the semantic data can be further used for visual related post-processing, it will bring the user A deeper visual experience.

Accordingly, it is an object of the present invention to provide a semantic-based visual effect generation system.

Therefore, the semantic-based visual effect generation system of the present invention comprises a semantic processing module and a visual processing module. The semantic processing module includes a semantic classification unit; the semantic classification unit is configured to receive a semantic data group related to a source material, and perform semantic/category correspondence classification on the semantic data group to The content is correspondingly classified to the at least one semantic category, thereby generating a semantic category association including the semantic category, wherein the source material includes an image material, and at least one of a sound material and a text material. The visual processing module includes a visual parameter generating unit and a visual post-production unit; the visual parameter generating unit is configured to receive the semantic category association and collect the visual parameter group according to the semantic category to obtain a visual parameter group. The unit is configured to generate at least one visual effect according to the visual parameter set, and display the image material of the source material in combination with the visual effect.

Another object of the present invention is to provide a program product that stores a semantic-based visual effect generating program.

Therefore, the program product storing the semantic-based visual effect generating program of the present invention can be loaded into an electronic device to perform the semantic processing module based on the semantic generating visual effect system, and the function of the visual processing module. .

The effect of the present invention is that the visual processing module and the visual processing module generate the visual effect reflecting the connotation of the source data, and combine the image data of the source material with the visual effect to display Give users a deeper visual experience.

The foregoing and other technical aspects, features and advantages of the present invention will be apparent from the following description of the preferred embodiments.

Referring to FIG. 1 , a preferred embodiment of the semantic-based visual effect generating system of the present invention includes a source data classification module 1 , a semantic processing module 2 coupled to the source data classification module 1 , and coupled to the A visual processing module 3 of the semantic processing module 2. In the preferred embodiment, the semantic-based visual effect generation system is implemented in a software manner, and the implementation aspect thereof is a program product for storing a semantic-based visual effect generation program, when an electronic device (for example, a computer processing) After the program is loaded and executed, the source data classification module 1, the semantic processing module 2, and the function of the visual processing module 3 can be completed.

The source data classification module 1 is configured to receive a source data and classify the source data, wherein the source material includes an image data, and at least one of a sound data and a text data, the image data may be A still image, or a video that includes a series of images. In the preferred embodiment, the source material is a digital multimedia material, which includes an image data, a sound data, and a text data.

The semantic processing module 2 includes an image analyzer 21, a sound analyzer 22, a text analyzer 23, and a semantic classification unit 24 coupled to the image, sound and text analyzers 21-23, coupled to the image. An image object database 25 of the analyzer 21, a keyword database 26 coupled to the text analyzer 23, and a semantic/category database 27 coupled to the semantic classifying unit 24.

The image analyzer 21, the sound analyzer 22, and the text analyzer 23 are configured to receive the classified source data, and perform semantic analysis on the image, sound and text data respectively to obtain related information about the source material. A semantic data set. The processing performed by the image analyzer 21, the sound analyzer 22, and the text analyzer 23 is further described below.

The image analyzer 21 is configured to analyze the image data of the source data to obtain the semantic data group, wherein the image analyzer 21 determines a brightness value corresponding to the image for one image of the image data. And comparing the value, and comparing the image object according to the set of image objects stored in the image object database 25 to extract at least one key object from the image; the semantic data group includes The brightness value corresponding to the image, the comparison value, and the important object. In the preferred embodiment, the image analyzer 21 obtains an average value of the luminance of all pixels in the image as the brightness value; and obtains a maximum brightness and a brightness among all the pixels. A difference value of the minimum brightness is taken as the comparison value. The image object collection is pre-established and stored in the image object database 25, and the image object collection includes image objects commonly used in daily life, such as vehicles, people, building facilities, etc., and the image object database 25 can also be Expand and update content as needed for use.

The sound analyzer 22 is configured to analyze the sound data of the source material to obtain the semantic data set, wherein the sound analyzer 22 is corresponding to one of the audio segments of the sound data. At least one frequency (frequency) and at least one amplitude of the sound segment; the semantic data set further includes the frequency corresponding to the sound segment and the amplitude. In the preferred embodiment, the frequency is higher in the sound segment, and the amplitude is greater in the sound segment.

The text analyzer 23 is configured to analyze the text data of the source material to obtain the semantic data group, wherein the text analyzer 23 is based on a keyword (word segmentation) of the text material. The key word) is compared to extract a representative at least one keyword from the text field; the text analyzer 23 further determines the pair according to the duration of the text field and a total number of words. A text speed of the text field; the semantic data set also includes the keyword corresponding to the text field and the text speed. In the preferred embodiment, the keyword set is pre-established and stored in the keyword database 26, and the keyword set includes common keyword pools, such as proper nouns, place names, etc., the keyword database. 26 can also expand and update content as needed for use.

The semantic classification unit 24 is configured to perform semantic/category correspondence classification on the semantic data group to classify the content of the semantic data group into at least one semantic category, thereby generating a semantic category association including the semantic category. (union). In the preferred embodiment, the semantic classification unit 24 performs semantic/category correspondence classification on the semantic data group according to a semantic/category association group, and the generated semantic category association includes a complex semantic category. The semantic/category association group is pre-established and stored in the semantic/category database 27, and the semantic/category association group includes a luminance value/category correspondence, a comparison value/category correspondence, and an image object/ Category correspondence, a frequency/category correspondence, an amplitude/category association, a keyword/category association, and a text speed/category association; the semantic/category association is established by counting a large number of digital multimedia materials It is mainly used to map the content of the semantic data group to a semantic category with a specific meaning. The semantic/category association group is shown in Table 1 below.

The visual processing module 3 includes a visual parameter generating unit 31 and a visual post-production unit 32 coupled to the visual parameter generating unit 31.

The visual parameter generating unit 31 is configured to receive the semantic category association and generate a visual parameter group according to the semantic categories of the semantic category association. The visual parameter generating unit 31 is configured according to a semantic category/adjustment parameter association, a semantic category/subtitle parameter association, a semantic category/text parameter association, and each semantic category of the semantic category association. The visual parameter set. In the preferred embodiment, the semantic category/adjustment parameter association, the semantic category/subtitle parameter association, and the semantic category/text parameter association are complex judgment conditions integrated into the program, but the semantic category/adjustment parameter association, The semantic category/subtitle parameter association, and the semantic category/text parameter association may also be pre-established in a database (not shown), and is not limited to the preferred embodiment; the semantic category/adjustment parameter association is based on The semantic category is associated with each semantic category, and its intersection or union, correspondingly determining the visual parameter set for adjusting a single image, which includes a special effect filter (filter), a background contrast adjustment parameter, and a a background brightness adjustment parameter group; the semantic category/subtitle parameter association is based on the semantic category of the semantic category, and an intersection or a combination thereof, correspondingly determining the visual parameter group related to a semantic enhanced subtitle, including At least one of a font size, a color, a font, and a subtitle effect corresponding to semantically enhanced subtitles.

The visual post-production unit 32 is configured to generate at least one visual effect according to the visual parameter set, and output and display the image data of the source material together with the visual effect, wherein the visual effect may be a dynamic visual effect or a static Visual effect. The visual effect includes a single image special effect for performing image adjustment processing on the image data, the semantic enhanced subtitle for superimposing and displaying the image data, and a text adjustment for adjusting the text data. At least one of the special effects. It is worth mentioning that the detailed implementation of the visual post-production unit 32 is well known to those skilled in the art, and there are many softwares related to visual post-production, and therefore will not be described here.

Referring to FIG. 1 , FIG. 2 , and FIG. 3 , corresponding to the above preferred embodiment, a semantic-based visual effect generating method and an application example are combined to use the source data classification module 1 and the semantic processing module 2 . And the interaction of the visual processing modules 3 with each other is explained. The semantic-based visual effect generating method includes the following steps.

As shown in step S41, the source data classification module 1 classifies a source data to obtain an image data, a sound data, and a text data.

In this application example, the source data is a digital audio and video material with subtitles, one of the image 5 of the image data and one of the text fields 6 of the text data is as shown in FIG. 3; wherein the text field 6 is a segment The content of the subtitles, that is, "Jianhushan Playground will be superimposed to show the gunpowder blast show", one of the sound segments of the sound data is a sound corresponding to the text field 6.

As shown in step S42, the image analyzer 21, the sound analyzer 22, and the text analyzer 23 of the semantic processing module 2 perform semantic analysis on the image, sound, and text data classified in step S41, respectively. Get a semantic data set related to the source material.

In this application example, the semantic data set includes: a luminance value corresponding to the image 5 (assumed to be 135), a contrast value (assumed to be 90), and an important object 51 (assumed to be a Ferris wheel), Should have a frequency of the sound segment (assumed to be 350 Hz) and two amplitudes (assuming 70 decibels and 80 decibels respectively), multiple keywords (assumed to be Jianhushan, gunpowder, blast show), and a text speed (assumed to be 80) Word/min).

As shown in step S43, the semantic classification unit 24 of the semantic processing module 2 respectively determines the brightness value, the comparison value, the important object obtained in step S42 according to the semantic/category association group shown in Table 1. The frequency, the amplitude, the keywords, and the text velocity are correspondingly classified to the associated plural semantic category, and a semantic category association including the semantic categories is generated.

In this application example, the brightness value (135) is classified into a brightness category _3 indicating that the brightness of the image 5 is high brightness; the comparison value (90) is classified into a comparison category _5 indicating the image 5 The contrast is medium to high contrast; the important object 51 (Ferris wheel) is classified into an image object category_2, which indicates that the image 5 is related to the playground; the frequency (350 Hz) is classified into a frequency category _6, which a female voice indicating that the sound segment is high; the amplitudes (70 decibels and 80 decibels) are also classified into an amplitude category _7 indicating that the sound segment is at a high volume; the keywords (Jianhushan, gunpowder, blasting show) ) respectively classified into a keyword category _7, a keyword category _10, and a keyword category _3, which respectively indicate a playground name, a proper noun, and an activity name; the text speed (80) The word/minute is classified into a text speed category _4, which indicates that the speaking speed is fast.

Further, it is assumed that "high brightness" means that the image of the image 5 is happy, "middle high contrast" means that the image of the image 5 is happy or lively, and "related to the playground" represents the image 5 according to the statistics performed in advance. The image is happy, the emotion represented by "sorghum" is excitement or excitement, the emotion represented by "high volume" is excitement, the situation represented by "playground name, proper noun, and activity name" is lively, "fast speaking speed" The emotions represented are excited or excited. The semantic category union generated by the semantic classification unit 24 is expressed as follows: {luminance category _3, contrast category _5, image object category_2} + sword lake hill {keyword category _7} + gunpowder {amplitude category _7 , keyword category _10} + blast show {frequency category _6, amplitude category _7, keyword category _3}, from which it can be seen that the semantic category association can reflect the connotation of the source material.

As shown in step S44, the visual parameter generating unit 31 of the visual processing module 3 obtains the semantic category/adjustment parameter association, the semantic category/subtitle parameter association, the semantic category/text parameter association, and the step S43. The semantic categories of the semantic category are combined to obtain the visual parameter set.

In this application example, the visual parameter generating unit 31 determines the visual parameter group according to the semantic category/adjustment parameter association, and the brightness category_3, the comparison category_5, and the image object category_2. a background brightness adjustment parameter set for brightening the background of the image 5; the visual parameter generation unit 31 associates according to the semantic category/subtitle parameter, and the frequency category _6, the amplitude category _7, the keyword The category_7, the text speed category _4 is determined, and the obtained visual parameter group includes a font size of a large font, a color of orange, a font of bold shadow, and a subtitle effect as a flame effect; Furthermore, due to the semantic category, gunpowder {amplitude category_7, keyword category_10}+blasting show {frequency category_6, amplitude category_7, keyword category_3} indicates that the gunpowder and the blast show are not However, the keyword is read at a relatively high volume and a high frequency. Therefore, the visual parameter generating unit 31 selects the two keywords as a subtitle content; the visual parameter generating unit 31 closes according to the semantic category/text parameter. , And the keyword category _7 judge, the visual parameters obtained are: to be keyword categories _7 text (ie, Janfusun) set a parameter adjusted to increase the font of the text profile.

Referring to FIG. 1 , FIG. 2 , and FIG. 4 , as shown in steps S45 to S46 , the visual post-production unit 32 of the visual processing module 3 generates corresponding visual effects according to the visual parameter set obtained in step S44. The image data of the source material is combined and displayed together with the visual effects.

In this application example, the visual post-production unit 32 illuminates a background 52 of the image 5 for subsequent output and display according to the background brightness adjustment parameter set; the visual post-production unit 32 further determines the font size, the color, The font, the subtitle effect, and the subtitle content generate a semantic enhanced subtitle 7 as shown in FIG. 4, and the semantic enhanced subtitle 7 is superimposed on the image 5 for subsequent output and display; the visual post-production unit 32 further enlarges the font of the text 61 corresponding to the keyword category _7 according to the adjusted parameter group of the font; however, the text adjustment effect is not limited to the specific text of the text data. For font adjustment, adjustment or addition of the text data 6 may be performed. For example, if the semantic category associated with the text material 6 is indicated as a non-stereo word, the inconspicuous words may be crossed, Delete, or replace with a space.

Referring to FIG. 1 , FIG. 4 , and FIG. 5 , the subtitle effect generated by the visual parameter generating unit 31 may also be an animation subtitle effect, and the visual post-production unit 32 may generate the semantic enhanced subtitle with an animation effect according to the subtitle effect. 7, in the semantic enhanced subtitles 7 of FIG. 4 and FIG. 5, the shaded portion of the "blasting show" can be rotated back and forth; and the subtitle effect generated by the visual parameter generating unit 31 can also be a specific animation effect. For example, the flame effects that can be flickering in Figures 4 and 5 can be seen.

It should be noted that, in this example, the single image 5 is used for description. However, the visual rear unit 32 can perform similar processing on the video including a series of images, and is not limited to the example.

In summary, the semantic processing module 2 generates the semantic category association, and the visual processing module 3 generates the corresponding visual effect to perform visual enhancement processing on the source data, and then output; The visual effect that the user can see the source material at the same time and reflect the connotation of the source material does bring a deeper visual experience to the user, so the object of the present invention can be achieved.

The above is only the preferred embodiment of the present invention, and the scope of the invention is not limited thereto, that is, the simple equivalent changes and modifications made by the scope of the invention and the description of the invention are All remain within the scope of the invention patent.

1. . . Source data classification module

2. . . Semantic processing module

twenty one. . . Image analyzer

twenty two. . . Sound analyzer

twenty three. . . Text analyzer

twenty four. . . Semantic classification unit

25. . . Image object database

26. . . Keyword database

27. . . Semantics/category database

3. . . Visual processing module

31. . . Visual parameter generation unit

32. . . Visual post unit

S41~S46. . . step

5. . . image

51. . . Important object

52. . . background

6. . . Field

61. . . Text

7. . . Semantic enhanced subtitles

1 is a system diagram illustrating a preferred embodiment of a semantic-based visual effect generation system of the present invention;

2 is a flow chart illustrating a semantic visual effect generating method corresponding to a preferred embodiment of the present invention;

3 is a schematic diagram showing one image of an image data of a source material, and one of the text fields of a text material;

4 is a schematic diagram showing the image material of the source material combined with various visual effects according to a preferred embodiment of the present invention; and

FIG. 5 is a schematic diagram illustrating an animation subtitle effect in conjunction with FIG. 4.

1. . . Source data classification module

2. . . Semantic processing module

twenty one. . . Image analyzer

twenty two. . . Sound analyzer

twenty three. . . Text analyzer

twenty four. . . Semantic classification unit

25. . . Image object database

26. . . Keyword database

27. . . Semantics/category database

3. . . Visual processing module

31. . . Visual parameter generation unit

32. . . Visual post unit

Claims (13)

A semantic-based visual effect generation system, comprising: a semantic processing module, comprising a semantic classification unit, the semantic classification unit is configured to receive a semantic data group related to a source material, and use the semantic data group to perform semantics on the semantic data group / class corresponding classification, to classify the content of the semantic data group into at least one semantic category, thereby generating a semantic category association including the semantic category, wherein the source material includes an image data, and a sound At least one of the data and the text data; and a visual processing module comprising a visual parameter generating unit and a visual post-production unit, the visual parameter generating unit configured to receive the semantic category association, and according to the semantic The categorization is performed to obtain a visual parameter set, and the visual post-production unit is configured to generate at least one visual effect according to the visual parameter set, and display the image data of the source material in combination with the visual effect. The semantic-based visual effect generating system according to claim 1, wherein the semantic processing module further includes an image analyzer for analyzing the image data of the source material to obtain the semantic data group. The image analyzer obtains a brightness value corresponding to the image for the image of the image data, the set of semantic data includes the brightness value corresponding to the image, and the semantic classification unit is based on the pre-established A luminance value/category corresponds to the association, and the luminance value is correspondingly classified to the associated semantic category. The semantic-based visual effect generating system according to claim 1, wherein the semantic processing module further includes an image analyzer for analyzing the image data of the source material to obtain the semantic data group. The image analyzer obtains a contrast value corresponding to the image, the semantic data set includes the contrast value corresponding to the image, and the semantic classification unit is based on the pre-established A comparison value/category corresponds to the association, and the comparison value is correspondingly classified to the associated semantic category. The semantic-based visual effect generating system according to claim 1, wherein the semantic processing module further includes an image analyzer for analyzing the image data of the source material to obtain the semantic data group. The image analyzer is configured to perform at least one important object from the image according to a pre-established image object set, wherein the semantic data set includes the important object. The semantic classification unit classifies the important object corresponding to the associated semantic category according to a pre-established image object/category correspondence. The semantic-based visual effect generating system of claim 1, wherein the semantic processing module further comprises a sound analyzer, the source data comprising the image data and the sound data, and the sound analyzer is used for And analyzing the sound data to obtain the semantic data group, wherein, for one sound segment of the sound data, the sound analyzer determines at least one frequency corresponding to the sound segment, and the semantic data group includes a corresponding sound segment The frequency classification unit is configured to classify the frequency into the associated semantic category according to a pre-established frequency/category correspondence. The semantic-based visual effect generating system of claim 1, wherein the semantic processing module further comprises a sound analyzer, the source data comprising the image data and the sound data, and the sound analyzer is used for The sound data is analyzed to obtain the semantic data set, wherein, for one sound segment of the sound data, the sound analyzer determines at least one amplitude corresponding to the sound segment, and the semantic data set includes the corresponding sound segment The amplitude of the semantic classification unit is based on a pre-established amplitude/category correspondence, and the amplitude is correspondingly classified to the associated semantic category. The semantic-based visual effect generating system according to claim 1, wherein the semantic processing module further includes a text analyzer, the source data includes the image data and the text data, and the text analyzer is used to The text data is analyzed to obtain the semantic data set, wherein, for one of the text fields of the text data, the text analyzer compares according to a pre-established set of keywords to extract at least one from the text field. The keyword, the semantic data group includes the keyword corresponding to the text field, and the semantic classification unit classifies the keyword into the belonging semantic category according to a pre-established keyword/category correspondence. The semantic-based visual effect generating system according to claim 1, wherein the semantic processing module further includes a text analyzer, the source data includes the image data and the text data, and the text analyzer is used to The text data is analyzed to obtain the semantic data group, wherein, for one of the text fields of the text data, the text analyzer obtains a text speed corresponding to the text field, and the semantic data group includes the text corresponding to the text field Speed, the semantic classification unit classifies the text speed correspondingly to the associated semantic category according to a pre-established text speed/category correspondence. The semantic-based visual effect generating system according to claim 1, wherein the visual effect is a single image special effect of performing image adjustment processing on the image data of the source material, and the visual parameter generating unit is based on A pre-established semantic category/adjustment parameter association, and the semantic category collection, to obtain the visual parameter set. The semantic-based visual effect generating system according to claim 1, wherein the visual effect is a semantic enhanced subtitle for superimposing and displaying the image data of the source material, the visual parameter generating unit According to a pre-established semantic category/subtitle parameter association, and the semantic category collection, the visual parameter group is obtained. The semantic-based visual effect generating system according to claim 10, wherein the visual parameter set includes at least one of a font size, a color, a font, and a subtitle effect corresponding to the semantic enhanced subtitle. . The semantic-based visual effect generating system according to claim 1, wherein the source data includes the image data and the text data, and the visual effect is a text adjustment effect for adjusting the text data. The visual parameter generating unit is based on a pre-established semantic category/text parameter association, and the semantic category is combined to obtain the visual parameter group. A program product for storing a semantic-based visual effect generating program, which, when loaded into an electronic device and executed, can complete the semantic processing module described in any one of the patent applications 1 to 12, and the visual processing The function described by the module.