TWI751571B - Video playback system and environment atmosphere adjusting method - Google Patents

Abstract

A video playback system and an environment atmosphere adjusting method are provided. In the method, a shrinking process is performed on an image frame, to obtained a shrank image frame. A foreground division process is performed on the shrank image, to obtain shrank foreground of interest. The foreground division process includes distinguishing a corresponding shrank foreground of interest according to brightness of multiple pixels of each shrank image frame. A representative color of the foreground of interest is determined. Then, the representative color is provided to the light, so that the light emits with the representative color, and the display plays the image frame. Accordingly, the visual experience of the viewer can be improved.

Description

Environment atmosphere adjustment system and environment atmosphere adjustment method

The present invention relates to an image and environment analysis combined with light source control technology, and in particular, to an environment atmosphere adjustment system and an environment atmosphere adjustment method.

In recent years, people can apply for a variety of video on demand (Video On Demand, VOD) services such as Netflix, Disney+, or MOD, and then they can watch the movies or albums they want to watch at home. On the other hand, the resolutions supported by various types of displays (eg, Liquid-Crystal Display (LCD), Light-Emitting Diode (LED) display, projector, etc.) Size (for example, more than 50 inches) specifications are gradually becoming more affordable. If they want to further enhance the viewing experience, some people will also buy surround sound, which can easily turn the living room into a small movie theater. In addition, some people will turn off the light sources in their environment to create a movie theater atmosphere. However, there is no correlation between the atmosphere of the environment and the content of the video, making such an atmosphere rather uninteresting.

In view of this, embodiments of the present invention provide an ambient atmosphere adjustment system and an ambient atmosphere adjustment method, which dynamically change the atmosphere based on changes in the light source of the video to be played.

The environment atmosphere adjustment method according to the embodiment of the present invention includes the following steps: performing thumbnail processing on an image frame to obtain a thumbnail image frame. A foreground segmentation process is performed on the image frame of the thumbnail to obtain an interesting foreground of the thumbnail. The foreground segmentation process includes: distinguishing the foreground of interest of the thumbnail according to the brightness of a plurality of pixels in the image frame of the thumbnail. According to the interest foreground of the thumbnail, the representative color of the thumbnail is determined. Provides the thumbnail's representative color to the light source. Let the light source emit light of the representative color of the thumbnail. Instruct the monitor to play the video screen.

The ambient atmosphere adjustment system of the embodiment of the present invention includes (but is not limited to) a display, a light source, and a processor. The display is used to play video images. The light source is used to emit light of various colors. The processor is connected to the display and the light source. The processor performs thumbnail processing on the image frame to obtain a thumbnail image frame, and distinguishes the interest foreground of the thumbnail according to the brightness of the pixels of the multiple thumbnails in the thumbnail image frame. The processor determines the representative color of the thumbnail in the foreground of interest of the thumbnail, provides the representative color of the thumbnail to the light source, makes the light source emit light of the representative color of the thumbnail, and causes the display to display the image.

Based on the above, the environmental atmosphere adjustment system and the environmental atmosphere adjustment method according to the embodiments of the present invention determine the representative color according to the interest foreground segmented in the thumbnail image, and control the light source to use the corresponding representative color in the process of playing the image. It emits light, so that the image screen can be related to the ambient atmosphere. Thereby, a novel atmosphere rendering experience can be provided, thereby enhancing the visual experience of the viewer.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, the following embodiments are given and described in detail with the accompanying drawings as follows.

FIG. 1 is a block diagram of components of an ambient atmosphere adjusting system 100 according to an embodiment of the present invention. Referring to FIG. 1 , an ambient atmosphere adjustment system 100 (or a video playback system) includes (but is not limited to) an image providing device 110 , an ambient sensor 130 , a display 150 , a light source 170 and a processor 190 .

The image providing device 110 may be a set-top box, a TV box, a smart phone, a tablet computer, a desktop/notebook computer, or a server. In one embodiment, the image providing device 110 is used to provide an image stream or a single image frame. The video stream or video frame may be downloaded from the network, transmitted by an external device (eg, a USB flash drive, a hard disk or a smart phone, etc.), or generated by the video providing device 110 by itself. In some embodiments, the image frame is one image frame of a plurality of image frames in the image stream.

The environment sensor 130 includes (but is not limited to) an image capturing device 131 and a microphone 133 . The image capturing device 131 can be a camera, a video camera or an image sensing element, and is used to capture an image of the environment in the environment. The microphone 133 may be of a dynamic type, a condenser type, an electret condenser (Electret Condenser), a Micro Electrical-Mechanical System (MEMS), etc., and is used to record ambient sounds in the environment. .

In some embodiments, the environmental sensor 130 may be a sensor for sensing temperature, brightness, humidity, and other environmental factors.

The display 150 can be a liquid crystal display (Liquid-Crystal Display, LCD), or a light-emitting diode (Light-Emitting Diode, LED) display, or can be any type of liquid crystal display, digital light processing technology (Digital Light Processing, DLP), etc. Projector. In one embodiment, the display 150 is used to play the video stream or the video frame provided by the video providing device 110 .

The light source 170 may be a multi-color LED formed by one or more LEDs. For example, by controlling the respective brightness of the red LED, the green LED and the blue LED, various light combinations of different colors can be formed. In one embodiment, the light source 170 is configured to emit light of multiple colors, and only emit light of one color at the same time, but may emit light of another color at another time (eg, three primary colors (ie, RGB) of light. another color with a different intensity). In addition, the light source 170 can receive a setting command of a specific RGB value and/or brightness, and emit light according to the specific RGB value and/or brightness. That is, it emits light of the color of this RGB value and the brightness of this lightness.

In one embodiment, the light source 170 is formed by LEDs arranged in a ring shape, so that the light emitted by the light source 170 can reach more areas in the environment.

It should be noted that the embodiments of the present invention do not limit the quantity, color, arrangement and arrangement of the LEDs in the light source 170 . In addition, the number of light sources 170 may also exceed one.

The processor 190 may be a processing unit such as a CPU, a microcontroller, a chip, an Application Specific Integrated Circuit (ASIC), or a Field Programmable Gate Array (FPGA). The processor 190 is wired or wirelessly connected to the image providing device 110 , the environment sensor 130 , the display 150 and the light source 170 . In some embodiments, at least two of the environmental sensor 130, the display 150, the light source 170, and the processor 190 may be integrated into a single device.

In this embodiment, the processor 190 may receive the image stream or image frame provided by the image providing device 110 and the sensing result (eg, ambient sound, ambient image, etc.) of the environment sensor 130, and control the display 150 to play the image , and control the light source 170 to emit light with a specific color and brightness, and its detailed operation will be described in detail in subsequent embodiments.

It is worth noting that the light emitted by the light source 170 can be seen by the viewer. When the viewer is watching the video stream or the video frame, it is also affected by the light emitted by the light source 170 . Lights of a specific color and/or brightness create different ambient ambiences. If the light-emitting operation of the light source 170 can be linked to the content of the video stream or the video frame, it will help to improve the viewing experience of the viewer.

In order to help understand the operation flow of the embodiment, the following description will be combined with the devices and mechanism components of the environment adjustment system 100 in FIG. 1 . FIG. 2 is a flow chart of a method for adjusting environmental atmosphere according to an embodiment of the present invention. Referring to FIG. 2 , the processor 190 receives an image stream or an image frame provided by the image providing device 110 . It should be noted that, the frame spacing of the plurality of image frames in the aforementioned video stream may be 10, 30, or 60 frames, etc., or the sampling frequency may be 1, 2, or 6 image frames per second. That is, only part of the video frame is obtained from the video stream. Additionally, the frame spacing or sampling frequency may be fixed or may be varied. Thereby, calculation efficiency can be achieved, and a more comfortable light source change speed can be provided.

In order to improve the subsequent processing speed, the processor 190 performs image pre-processing on the image frame. The processor 190 may perform thumbnail processing (eg, reducing the resolution) on the original image frame (eg, the R (red) G (green) B (blue) image) to generate a thumbnail image frame (step S210 ). . For example, the processor 190 generates a thumbnail image with a resolution of 320*192 after the original image with a resolution of 1280*768 is thumbnailed to one-sixteenth.

In some embodiments, the processor 190 further performs gray-scale processing on the image frame of the thumbnail to obtain a gray-scale image of the thumbnail. For example, the processor 190 calculates the brightness of each pixel in the image frame as the sum of the red intensity value of 0.299, the green intensity value of 0.857, and the blue intensity value of 0.114. The processor 190 can obtain the brightness corresponding to each pixel from the grayscale image.

FIG. 3 is an example of an image screen illustrating a thumbnail image. Please refer to FIG. 3 , which shows a thumbnail image screen. FIG. 4 is a grayscale image illustrating an example of a thumbnail. Please refer to FIG. 4 , which shows a grayscale image of the image frame based on the thumbnail of FIG. 3 .

Next, the processor 190 performs foreground segmentation processing on the image frame of the thumbnail, so as to obtain the interesting foreground of the thumbnail (step S220 ). It should be noted that most of the image frames are composed of foreground (eg, people, animals, or objects, etc.) and background (eg, sky, mountain, or sea, etc.). The viewer usually pays more attention to the foreground content, so the embodiment of the present invention will first segment the foreground that the viewer is more interested in (hereinafter collectively referred to as the interesting foreground). There are many methods for foreground segmentation processing. In the embodiment of the present invention, the processor 190 can distinguish the foreground of interest of the thumbnail according to the brightness of a plurality of pixels in the image frame of the thumbnail. Specifically, lightness can be referred to as gray scale or gray value. Generally speaking, there is a significant difference in brightness between the foreground and background in an image. The embodiment of the present invention divides the foreground based on such a difference.

After obtaining the grayscale image, the processor 190 may perform foreground segmentation processing on the grayscale image. 5A is a flowchart of a foreground segmentation process according to an embodiment of the present invention. Referring to FIG. 5A , the processor 190 can make the brightness of the thumbnail image too low (eg, lower than 75, 60, or 50, etc.) and/or too high (eg, higher than 225, 230, or 250, etc.) ) and other extreme values are filtered out (step S226).

In one embodiment, the processor 190 may set a high brightness threshold (eg, 75, 60, or 50, etc.) to filter one or more pixels in the thumbnail image whose brightness is higher than the high brightness threshold, And keep the remaining pixels in the thumbnail image whose brightness is lower than the high brightness threshold (ie, the remaining pixels that are not filtered are reserved pixels).

In another embodiment, the processor 190 may set a low brightness threshold (eg, 225, 230, or 250, etc.) to filter one or more pixels whose brightness is lower than the low brightness threshold in the thumbnail image. , and keep the remaining pixels in the thumbnail image whose brightness is higher than the low brightness threshold. For example, a grayscale image includes 61,440 pixels, with 43,008 remaining after filtering.

In yet another embodiment, the processor 190 may count the number of pixels of each brightness in a single image frame. If the number of pixels whose brightness is higher than the high brightness threshold is greater than the filtering threshold, the processor 190 performs high-pass filtering on these pixels. For example, filter out pixels with lightness higher than 225. If the number of pixels whose brightness is lower than the high brightness threshold is greater than the filtering threshold, the processor 190 performs low-pass filtering on these pixels. For example, filter out pixels with lightness below 75. In another embodiment, the processor 190 may perform two-stage processing of low-pass filtering and high-pass filtering. For example, first filter out pixels with lightness less than 100, and then filter out pixels with lightness greater than 200. In yet another embodiment, the processor 190 may perform mid-pass (band-pass) filtering processing. That is, pixels whose lightness is between the low lightness threshold and the high lightness threshold are reserved. For example, keep pixels with lightness between 50 and 220.

FIG. 5B is an example illustrating extreme value filtering. Referring to FIG. 5B , after the gray-scale image is low-pass filtered, the black background will be filtered, and the retained gray-scale image is obviously only the strawberry image.

It should be noted that the high-brightness threshold is higher than the low-brightness threshold, but the values of the high-brightness threshold and the low-brightness threshold can still be determined according to actual needs and may be changed according to the number of corresponding pixels.

Next, the processor 190 determines a critical brightness value for converting the binarized image according to the brightness of one or more remaining remaining pixels in the thumbnail image frame. These remaining remaining pixels are the unfiltered pixels in step S226. Since the brightness distribution in the image frames of different thumbnails may be different, a single critical brightness value may not be applicable to all the image frames. Therefore, the embodiment of the present invention dynamically changes the critical brightness value based on the brightness distribution of the current image frame (step S227 ). There are many algorithms for adaptive critical lightness value, such as Otsu (OTSU) algorithm (which can accurately distinguish two gray-scale pixel groups with the greatest difference), pixel gray value averaging (that is, a single image The sum of the brightness of all the pixels in the pixel is divided by the number of all the pixels, and the calculation is fast) algorithm, or Gaussian algorithm, etc.

Next, the processor 190 may convert the grayscale image of the thumbnail into a binarized image of the thumbnail (step S228 ). Specifically, the processor 190 may perform a binarization process on the grayscale image of the thumbnail according to the set threshold brightness value, so as to obtain the binarized image of the thumbnail. The image binarization process is to change the pixels whose brightness is higher than the critical brightness value to the maximum brightness value (for example, 255), and change the pixels whose brightness is lower than the critical brightness value to the minimum brightness value (for example, 0), so as to implement binarization. That is, there are only maximum and minimum values. For example, if the critical lightness value is 154, pixels with lightness less than 154 are set to black, and pixels with lightness greater than 154 are set to white.

FIG. 5C is an example illustrating image binarization. Referring to FIG. 5C , after the binarization process, the strawberry image in the binarized image will be more prominent.

Next, the processor 190 may perform intersection processing on the image frame of the thumbnail and the binarized image of the thumbnail, so as to obtain an interesting foreground of the thumbnail. Those pixels with a lightness below the critical lightness value will be considered as pixels in the foreground of interest. That is, pixels with smaller luminance values in the binarized image form the foreground of interest. The processor 190 can compare the same position (ie, the intersection of positions) of the image frame (with color) of the original thumbnail according to the positions of those pixels in the binarized image that are regarded as the foreground of interest (ie, pixels with smaller luminance values) ), thereby segmenting an image with a color foreground of interest from the original thumbnail image.

It should be noted that, in some embodiments, step S226 and/or step S227 may be ignored. That is, do not filter extreme values or use a fixed critical lightness value.

Referring to FIG. 2 , after determining the interest foreground, the processor 190 may determine the representative color of the thumbnail according to the interest foreground of the thumbnail (step S230 ). Specifically, the interest foreground of the thumbnail may still include many colors, and the processor 190 will further select a single representative color representing the image frame of the entire thumbnail from these colors.

In one embodiment, the processor 190 may divide several interest pixels in the foreground of interest in the thumbnail into several color groups, and determine the color of the representative color according to the number of pixels in these color groups. Specifically, the interest pixels refer to those pixels corresponding to the interest foreground. In the embodiment of the present invention, the representative color is selected based on statistical cluster analysis.

There are many examples of representative color determination. FIG. 6 is a flowchart of representative color determination according to an embodiment of the present invention. Please refer to FIG. 6 , there are many kinds of clustering algorithms. The processor 190 can use a clustering algorithm such as K-Mean, K-Medoid, etc. (fast convergence speed and less tuning parameters) to distinguish those pixels of interest into a specific number of color groups (step S610 ). ). The colors of the pixels of interest in the same color group may be relatively close or only have a single color, and each color group corresponds to one representative color. If a color group includes interest pixels of different colors, the average, median, or mode of the RGB values of those interest pixels in the same color group can be used as the RGB of the representative color of this color group value. The processor 190 may calculate the number of pixels in the color groups of the thumbnails (ie, the number of pixels included in the color groups of those thumbnails) (step S630 ), and sort the color groups of the thumbnails according to the number , and the representative color of the color group of the thumbnail with the largest number of pixels is selected as the final representative color of the thumbnail (step S650 ). For example, the RGB values of the representative color with the largest number of pixels are 128 for red, 170 for green, and 50 for blue.

FIG. 7 is an example illustrating the determination of the representative color. Referring to FIG. 7 , the interest foreground of the thumbnail (ie, the image of strawberry) can be obtained by performing intersection processing on the image frame of the colored thumbnail and the binarized image of the thumbnail. The strawberry image has the largest amount of red, so red can be used as the final representative color.

It should be noted that, in step S650, the color group with the largest number of pixels is selected, but in other embodiments, the color group with the second largest number of pixels, the third largest number of pixels, or other sorted color groups may also be selected.

FIG. 8 is a flowchart of representative color determination according to another embodiment of the present invention (the advantage is that the amount of computation is less, the computation speed can be increased, and the influence of ambient light can be reduced). Referring to FIG. 8 , the processor 190 converts the interest pixels of those thumbnails in the foreground of interest from the three primary color light (ie, RGB) color space to the Hue Saturation Lightness (HSV) color space (step S810). For example, the processor 190 may count the number of pixels of each RGB value in the pixel of interest to form RGB color space data. Next, the processor 190 converts the coordinates in the RGB color space to the coordinates in the HSV color space, and determines the number of pixels of each HSV value based on the previously counted number of pixels to form HSV color space data.

The processor 190 may define a plurality of chromaticity blocks of thumbnails according to a plurality of predetermined chromaticities for the HSV color space, and these chromaticity blocks will be used as color groups of the aforementioned thumbnails. That is, the color groups of the thumbnails are divided into the HSV color space.

FIG. 9 is a flowchart of chroma block determination according to an embodiment of the present invention, and FIG. 10A is an example illustrating a chroma block 1011 . Referring to FIG. 9 and FIG. 10A , the processor 190 provides a chromaticity statistics container (step S910 ) (the chromaticity statistics container 1010 shown in FIG. 10A , that is, the top viewing angle of the HSV color space), and sets the chromaticity statistics container according to the predetermined color. The degree unit is divided into a plurality of chroma blocks from the top (brightest) to the bottom (darkest) (step S930 ).

The predetermined chrominance unit will determine the size of the chrominance block. The predetermined chromaticity unit may be other equal angles (for example, 45 degrees, 15 degrees, etc.), or may be unequal angle divisions (for example, a certain chromaticity block is 337.5 degrees to 22.5 degrees, and another chromaticity block is 22.5 to 37.5 degrees, and another chroma block is 37.5 to 82.5 degrees, etc.). As shown in FIG. 10A , eight chrominance blocks 1011 of the same size are formed by dividing the chrominance statistics container 1010 into eight equal parts and equal angles. FIG. 10B is another example illustrating the chroma block 1013 . Referring to FIG. 10B , the different chromaticity blocks 1013 are formed by unequal angle division of the chromaticity statistics container 1010 .

In addition, the processor 190 can further define a plurality of chroma and lightness blocks of the thumbnails according to a plurality of predetermined chroma and a plurality of predetermined lightness for the HSV color space. FIG. 11 is a flowchart of determination of chrominance and luminance blocks according to an embodiment of the present invention, and FIG. 12A is an example illustrating the second chrominance block 1211 . 11 and FIG. 12A , the processor 190 provides a chroma and lightness statistics container (step S1110 ) (the chroma and lightness statistics container 1210 shown in FIG. 12 , in terms of the perspective of the HSV color space), and converts the color The luminance and lightness statistics container 1210 is divided into a plurality of second chromaticity blocks (color groups as thumbnails) from the top (brightest) to the bottom (darkest) according to another predetermined chromaticity unit (step S1130 ). The second chrominance block is formed by further dividing each chrominance block, and the predetermined chrominance unit of each chrominance block is larger than the predetermined chrominance unit of the second chrominance block therein. For example, the predetermined chromaticity unit of the chrominance block is 45 degrees of equal angle, and the predetermined chromaticity unit of the second chromaticity block includes 10 degrees, 15 degrees and 20 degrees of unequal angles. As shown in FIG. 12A , the chromaticity range of the second chrominance block 1211 is smaller than that of the chrominance block 1011 of FIG. 10A .

Next, the processor 190 divides each second chrominance block into a plurality of chrominance and luminance blocks according to the predetermined chrominance unit and the predetermined luminance unit respectively (step S1150 ). The chrominance corresponding to the chrominance and luminance blocks of each second chrominance block is limited by the chrominance range of the corresponding (or belonging) chrominance block. The predetermined chroma unit and the predetermined brightness unit will determine the size of the chroma and brightness block. It is assumed that chroma and lightness (ie, brightness) are values from 0% to 100%. The predetermined chroma unit and the predetermined lightness unit may be the same range (eg, equally spaced every 10%, 20%, or 30%). As shown in FIG. 12B , an example is illustrated for the chroma and lightness blocks 1213 , and the range of lightness S or the range of chroma V of each chroma and lightness block 1213 is the same. Alternatively, the predetermined chroma unit and the predetermined lightness unit may also include a variety of different ranges (eg, including 10%, 25%, or 45% equal spacing).

Returning to FIG. 8 , after these blocks are defined, the processor 190 can count the chrominance blocks of the thumbnails and the number of pixels of the chroma and luminance blocks of the thumbnails according to the HSV color space data (step S830 ). . That is, the total number of pixels in the chrominance blocks of each thumbnail and the total number of pixels in the chrominance and lightness blocks of each thumbnail are counted. It is worth noting that the number of pixels of the chroma block of the thumbnail is the sum of the number of pixels of the chroma and luminance blocks of all the thumbnails of all the second chroma blocks below it.

In this embodiment, the processor 190 determines the representative color (ie, color) of the thumbnails according to the number of pixels in the chroma blocks of those thumbnails. The processor 190 may sort the chroma blocks of the thumbnails according to the number of pixels (step S850 ), and determine whether the number of pixels of the chroma blocks of the thumbnails reaches a predetermined proportion (for example, a percentage of 30, 40 or 50 and the corresponding number) (step S870). For example, the processor 190 may sequentially read a predetermined number of chroma blocks and the number of pixels of the thumbnails, and determine whether the sum, average or any value of the number of pixels is greater than the predetermined proportion.

If the number of pixels in the chroma blocks of the one or more thumbnails reaches a predetermined proportion, the processor 190 may determine the pixel count corresponding to the second color block in the chroma blocks of the one or more thumbnails according to the The number of pixels in the chroma and lightness blocks of the thumbnail determines the representative color (step S871 ). For example, the processor 190 may define the chroma and lightness blocks of one or more thumbnails according to those predetermined chroma and a plurality of predetermined lightness. The processor 190 divides the color groups of the thumbnails according to the chroma and lightness blocks of the thumbnails. The processor 190 selects the chroma and lightness block of the thumbnail with the largest or the largest number of pixels according to the color group of the thumbnail and the number of pixels of the corresponding chroma and lightness block of the thumbnail, and determines the chroma and lightness block of the thumbnail to which it belongs The chroma range corresponding to the chroma block (as the selected chroma block and the chroma block of the thumbnail corresponding to the color group of the thumbnail). For another example, the processor 190 directly obtains the chroma block with the largest number of pixels (as the selected chroma block), and determines the corresponding chroma range. As another example, the processor 190 may select one or more chroma blocks from those predetermined number of chroma blocks as selected chroma blocks.

The processor 190 can determine the representative color of the thumbnail according to the selected color group of the thumbnail. FIG. 13 is a flowchart of representative color determination according to an embodiment of the present invention. Referring to FIG. 13 , the processor 190 can obtain the chroma values of the thumbnails from the selected chroma blocks (ie, the chroma blocks of the thumbnails corresponding to the color groups of the thumbnails) among those chroma blocks. , and according to the chroma and brightness block of the thumbnail corresponding to the selected chroma block, obtain the chroma and brightness value of the thumbnail (including the chroma of the thumbnail and the brightness value of the thumbnail) (step S1310) . For example, the chrominance value may be the chrominance mean, chrominance median, chrominance minimum, chrominance maximum, or any chrominance value in the chrominance range in the selected chrominance block. For another example, the chroma and lightness value may be the average value of chroma and lightness, the median value of chroma and lightness, the minimum value of chroma and lightness, the minimum value of chroma and lightness, the Cum brightness maximum or any value. For another example, the chroma and lightness values may be the average value of chroma and lightness, the median value of chroma and lightness, and the minimum value of chroma and lightness in the chroma and lightness blocks of the thumbnails in the selected chroma block. , the maximum value of chroma and lightness or any of them.

Next, the processor 190 may determine the representative color of the thumbnail of the image frame in the RGB color space according to the chrominance value and the chrominance and lightness value batches (step S1330 ). That is, the chromaticity value and the chrominance and lightness value are converted into RGB values, and the RGB values are used as the representative colors of the thumbnails.

Returning to FIG. 8 , if the number of pixels in the chroma blocks of none of the thumbnails reaches the predetermined proportion, the processor 190 may select the largest or the largest number of pixels from the chroma blocks of those thumbnails. The random number determines the representative color of the thumbnail (step S873). FIG. 14 is a flowchart of representative color determination according to another embodiment of the present invention. Referring to FIG. 14 , the processor 190 can obtain the selected chroma block of the thumbnail with the largest number of pixels from those chroma blocks (step S1410 ), and then the selected chroma block (eg, the above-mentioned picture) The second chrominance value is obtained from the one with the largest number of pixels), and the second chrominance and lightness value is obtained according to the chrominance and lightness block corresponding to the selected chrominance block (step S1430). For example, the second chrominance value may be the chrominance mean, chrominance median, chrominance minimum, chrominance maximum, or any chrominance value in the chromaticity range in the selected chrominance block. For another example, the second chroma and lightness value may be the average value of chroma and lightness, the median value of chroma and lightness, the minimum value of chroma and lightness, the average value of chroma and lightness of the thumbnail with the largest number of pixels, the The maximum value of chroma and lightness or any of them. For another example, the second chroma and lightness value may be the average value of chroma and lightness, the median value of chroma and lightness, the average value of chroma and lightness, the median value of chroma and lightness, the chroma and lightness of those thumbnails in the selected chroma block. Minimum value, maximum chroma and lightness value, or any of them.

Next, the processor 190 can determine the third chrominance value and the third chrominance and lightness value (as the color group of the thumbnail) according to the second chrominance value and the second chrominance and lightness value random numbers (step S1450 ). ). Specifically, the third chroma value and the third chroma-cum-lightness value are the chroma value and the chroma-cum-lightness value determined by random numbers in the chroma, chroma-cum-lightness range of the selected chroma block. For example, if the chrominance range of the selected chrominance block is 180 to 225, the second chrominance value is 202.5, and the third chrominance value is 202.5 plus a random number determined by -2.5 (200). For another example, the second chroma and lightness value corresponding to the chroma and lightness block with the largest number of pixels in the selected chroma block has a chroma value of 20% and a lightness of 50%, and the third chroma and lightness value is 20%. The lightness value is the result of 20% and 50% plus 2% and -10% determined by random numbers (22% chroma and 40% lightness).

Next, the processor 190 may determine the representative color of each image frame in the RGB color space according to the third chrominance value and the third batch of chrominance and lightness values (step S1470). That is, the third chromaticity value and the third chromaticity and lightness value are converted into RGB values, and the RGB values are used as the representative color.

It should be noted that, the embodiments of FIG. 8 and FIG. 14 may have various variations. For example, FIG. 8 directly determines the chrominance value, chroma and lightness value of the representative color of the thumbnail from the chrominance block with the largest number of pixels, without judging whether the predetermined proportion is reached. For another example, FIG. 14 directly selects the chroma and lightness block of the thumbnail with the largest number of pixels from the selected chroma blocks, and determines the representative of the thumbnail from the chroma and lightness block of the thumbnail. The chromaticity value and the chroma and lightness value of the color. For another example, the random value may be changed to a fixed value.

In addition to determining the representative color, in one embodiment, the brightness of the light source 170 can be further determined. FIG. 15 is a flowchart of determining the brightness of the light source 170 according to an embodiment of the present invention. Referring to FIG. 15 , the processor 190 can record and obtain the ambient sound of the environment through the microphone 133 in the environment where the video screen is scheduled to be played (step S1510 ), and quantify the audio data of the ambient sound (for example, by analogy The digital processing is performed to obtain the volume of the ambient sound) (step S1530 ), and then the brightness corresponding to the quantized decibel value linearly is determined (step S1550 ). This brightness can be used as the brightness of the light source 170 .

For example, Table (1) is an example of the linear correspondence between decibel values and brightness: Table (1) Decibel value brightness 0 0 60 128 greater than 120 255 When the volume of the ambient sound is larger, the brightness of the light source 170 is brighter, and when the volume of the ambient sound is smaller, the brightness of the light source 170 is darker.

It should be noted that the numerical values in Table (1) and their corresponding relationships are only used as examples, and are not intended to limit the embodiments of the present invention. In addition, in the embodiment of FIG. 15 , the corresponding brightness is obtained according to the amplitude of the ambient sound. However, in other embodiments, the processor 190 may also obtain the corresponding brightness according to the pitch of the ambient sound or a specific melody.

In another embodiment, the processor 190 can also obtain the ambient image of the environment (ie, the ambient sound captured in the environment where the display 150 plays the video stream) through the image capturing device 131, and determine the environment image according to the ambient image The brightness of the light source 170. For example, the brightness of the light source 170 is determined based on the average brightness of the ambient image, or the color of the maximum number of pixels (in a linear relationship or a specific mathematical relationship).

In other embodiments, according to the type of the environmental sensor 130 , the processor 190 may determine the corresponding representative color luminance based on different types of sensing results. Alternatively, the processor 190 may directly determine the brightness of the light source 170 according to the aforementioned first or third chroma and lightness values.

Returning to FIG. 2 , after the representative color of the thumbnail is determined, the processor 190 may provide the representative color of the thumbnail to the light source 170 (step S240 ), and make the light source 170 emit light of the representative color of the thumbnail (step S250 ), and The display 150 is synchronously instructed to play the video image (step S260). Specifically, in order to associate the content of the image screen with the ambient atmosphere, the embodiment of the present invention creates an ambient atmosphere by changing the color and/or brightness of the light source 170, and the color and/or brightness of the light source 170 is based on thumbnails It is determined by the representative color of the image screen. On the time axis, the light source 170 will change as the image frame corresponding to the thumbnail changes. For example, when the display 150 displays the image frames of the thirtieth to ninetieth frames, the light source 170 emits light in the representative color of the image frame based on the thumbnail of the thirtieth frame. The processor 190 can determine the frame currently being played on the display 150 and determine the representative color of the thumbnail corresponding to the frame, so that the light source 170 can emit light in the representative color of the thumbnail.

If the brightness of the light source 170 is additionally determined, the processor 190 can simultaneously determine the representative color of the thumbnail corresponding to the currently playing frame and the brightness of the current light source 170, so that the light source 170 can emit light with the representative color and brightness of the thumbnail. On the time axis, the light source 170 can be changed with the corresponding video frame and ambient sound/image.

FIG. 16 is a flow chart of light source control according to an embodiment of the present invention. Referring to FIG. 16 , within the same time interval, the processor 190 obtains the brightness according to the ambient sound/image (step S1610 ), and according to the RGB value of the representative color of the thumbnail (step S1620 ), and converts the representative color of the thumbnail After integrating with the brightness (ie, synchronizing on the time axis) (step S1630 ), the light source 170 can be controlled to emit light based on the representative color and brightness of the thumbnail (step S1650 ).

It should be noted that the foreground color in some continuous image frames may change too fast, causing the color of the light source 170 to change too fast and flicker. In order to avoid flickering of the light source 170, the embodiment of the present invention further proposes an improved solution.

FIG. 17 is a flowchart of light source update according to an embodiment of the present invention. Referring to FIG. 17 , the processor 190 can determine whether the first brightness (or the gray scale value, the brightness of the light source 170 ) of the representative color of the thumbnail corresponding to the image frame of the current frame is greater than the brightness threshold value (for example, 8, 10, or 50, etc.) (step S1710). If the first brightness is greater than the brightness threshold, the processor 190 further determines the color difference between the representative color of the thumbnail and the representative color of the previous thumbnail in the RGB color space or the HSV color space, or determines the representative color of the thumbnail Whether the lightness difference between the color and the representative color of the previous thumbnail is greater than a different color threshold (for example, 30, 50, or 80, etc.) (step S1730). The previous frame image frame is, for example, an image frame that is 10, 20, or 60 frames away from the current frame image frame. If the difference is greater than the different-color threshold, the processor 190 changes the light source 170 (ie, updates the color of the light source 170 ) according to the representative color of the thumbnail (step S1750 ), or. On the other hand, if the difference is not greater than the different color threshold or the first brightness is not greater than the brightness threshold, the processor 190 controls the light source 170 to maintain the representative color of the thumbnail corresponding to the previous frame (step S1770 ) (ie, maintain the light source 170 color). In this way, the frequency of color change of the light source 170 can be reduced, thereby improving the eye comfort of the viewer.

In another embodiment, the processor 190 may directly increase the sampling interval between the current image frame and the previous image frame. That is, the sampling frequency is decreased or the frame spacing is increased. For example, the representative color of the image frame that analyzes 6 thumbnails per second is reduced to the representative color of the image frame that analyzes 1 thumbnail per second.

To sum up, the environment adjustment system and the environment adjustment method according to the embodiments of the present invention analyze the video stream or image screen to be played to determine the interest foreground and the representative color of the thumbnail. In addition, the embodiment of the present invention also analyzes on-site environmental factors to determine the brightness of the light source. Then, in the process of playing the video stream or the video frame, the corresponding color (ie, the representative color) and brightness can be synchronously emitted through the light source. Thereby, the viewing experience of the viewer can be improved.

Although the present invention has been disclosed above by the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the scope of the appended patent application.

100: Ambient atmosphere adjustment system 110: Video providing device 130: Environment Sensor 131: Image capture device 133: Microphone 150: Monitor 170: light source 190: Processor S210~S260, S212~S214, S216~S218, S610~S650, S810~S873, S910~S930, S1110~S1150, S1310~S1330, S1410~S1470, S1510~S1550, S1610~S1650, S1710~S 1010: Chroma Statistics Container 1011, 1013: Chroma blocks 1210: Chroma and Brightness Statistics Container 1211: Second Chroma Block 1213: Chroma and Brightness Block S: chroma V: lightness

FIG. 1 is a block diagram of components of an ambient atmosphere adjusting system according to an embodiment of the present invention. FIG. 2 is a flow chart of a method for adjusting environmental atmosphere according to an embodiment of the present invention. FIG. 3 is an example of an image screen illustrating a thumbnail image. FIG. 4 is a grayscale image illustrating an example of a thumbnail. 5A is a flowchart of a foreground segmentation process according to an embodiment of the present invention. FIG. 5B is an example illustrating extreme value filtering. FIG. 5C is an example illustrating image binarization. FIG. 6 is a flowchart of representative color determination according to an embodiment of the present invention. FIG. 7 is an example illustrating the determination of the representative color. FIG. 8 is a flowchart of representative color determination according to another embodiment of the present invention. FIG. 9 is a flowchart of chroma block determination according to an embodiment of the present invention. Figure 10A is an example illustrating chroma blocks. FIG. 10B is another example illustrating chroma blocks. FIG. 11 is a flowchart of determination of chroma and lightness blocks according to an embodiment of the present invention. FIG. 12A is an example illustrating the second chroma block. FIG. 12B is an example illustrating chroma and lightness blocks. FIG. 13 is a flowchart of representative color determination according to an embodiment of the present invention. FIG. 14 is a flowchart of representative color determination according to another embodiment of the present invention. FIG. 15 is a flowchart of brightness determination of a light source according to an embodiment of the present invention. FIG. 16 is a flow chart of light source control according to an embodiment of the present invention. FIG. 17 is a flowchart of light source update according to an embodiment of the present invention.

S210~S260: Steps

Claims (25)

An environmental atmosphere adjustment method, comprising: Perform a thumbnail processing on an image frame to obtain a thumbnail image frame; Perform a foreground segmentation process on the image frame of the thumbnail to obtain a foreground of interest of the thumbnail, wherein the foreground segmentation process includes: Distinguish the interesting foreground of the thumbnail according to the brightness of the pixels of the thumbnails in the image frame of the thumbnail; Determine the representative color of a thumbnail according to the interest foreground of the thumbnail; providing the representative color of the thumbnail to a light source; causing the light source to emit light in the color representative of the thumbnail; and Make a display to play the video frame. The environment atmosphere adjustment method according to claim 1, wherein the image frame is an image frame among a plurality of image frames in an image stream. The environmental atmosphere adjustment method according to claim 1, wherein the step of distinguishing the interest foreground of the thumbnail according to the brightness of the pixels of the thumbnails in the image frame of the thumbnail includes: performing a grayscale process on the image frame of the thumbnail to obtain a grayscale image of the thumbnail; deriving the lightness of the pixels of the thumbnails from the grayscale image of the thumbnails; converting the grayscale image of the thumbnail into a binarized image of the thumbnail; and An intersection process is performed on the image frame of the thumbnail and the binarized image of the thumbnail to obtain an interesting foreground of the thumbnail. The ambient atmosphere adjustment method according to claim 3, before the step of converting the grayscale image of the thumbnail into the binarized image of the thumbnail, further comprising: Set a high brightness threshold; Filtering the image frame of the thumbnail, at least one of the pixels whose brightness is higher than the high brightness threshold; and The remaining pixels whose brightness is lower than the high brightness threshold in the image frame of the thumbnail image are retained. The environmental atmosphere adjustment method according to claim 3, wherein before the step of converting the grayscale image into the binarized image, the method further comprises: setting a low luminance threshold; and Filtering the image frame of the thumbnail, at least one of the pixels whose brightness is lower than the low brightness threshold; and The remaining pixels whose brightness is higher than the low brightness threshold in the image frame of the thumbnail image are retained. The environmental atmosphere adjustment method according to claim 4 or 5, further comprising: determining a critical luminance value for converting the binarized image according to the luminance of the remaining remaining pixels; and According to the critical brightness value, a binarization process is performed on the grayscale image of the thumbnail to obtain the binarized image of the thumbnail. The environmental atmosphere adjustment method according to claim 6, wherein the binarization process comprises an Otsu algorithm, an averaging algorithm, or a Gaussian algorithm. The environment atmosphere adjustment method according to claim 1, wherein the step of determining the representative color of the thumbnail in the foreground of interest of the thumbnail includes: The interest pixels of the plurality of thumbnails in the foreground of interest for the thumbnail are divided into color groups of the plurality of thumbnails; and The representative color of the thumbnail is determined according to the number of pixels in the color group of the thumbnails. The environment atmosphere adjustment method according to claim 8, wherein the step of distinguishing interest pixels of the thumbnails in the foreground of interest in the thumbnail into color groups of the thumbnails includes: converting the interest pixels of the thumbnails of the foreground interest of the thumbnail from an RGB color space to an HSV color space; and For the HSV color space, define chromaticity blocks of a plurality of thumbnails according to a plurality of predetermined chromaticities; For the HSV color space, according to a plurality of predetermined chroma and a plurality of predetermined lightness, define the chroma and lightness blocks of a plurality of thumbnails; and According to the intersection of the chrominance blocks of the thumbnails and the chrominance and lightness blocks of the thumbnails, the color groups of the thumbnails are defined. The ambient atmosphere adjustment method according to claim 9, wherein the step of determining the representative color of the thumbnails according to the number of pixels in the color groups of the thumbnails includes: count the number of pixels in the chroma blocks of the thumbnails; and The representative color of the thumbnail is determined according to the number of pixels in the chroma blocks of the thumbnails. The environmental atmosphere adjustment method according to claim 10, wherein the step of determining the representative color of the thumbnails according to the number of pixels of the chroma blocks of the thumbnails includes: It is judged whether the number of pixels of the chroma blocks of the thumbnails reaches a predetermined proportion. The environmental atmosphere adjustment method according to claim 11, wherein: If so, defining the chroma and brightness blocks of the thumbnails according to the predetermined chroma and a plurality of predetermined brightnesses for the chroma blocks of at least one of the thumbnails; dividing the color groups of the thumbnails according to the chroma and brightness blocks of the thumbnails; According to the color groups of the thumbnails, and the pixel numbers of the corresponding chroma and brightness blocks of the thumbnails, select a thumbnail color group with the largest number of pixels; and According to the selected color group of the thumbnail, the representative color of the thumbnail is determined. The environmental atmosphere adjustment method according to claim 11, wherein: If not, selecting the chroma block of a thumbnail with the largest number of pixels among the chroma blocks of the thumbnails; defining the chroma and luminance blocks of the thumbnails according to the predetermined chroma and a plurality of predetermined luminances; dividing the color groups of the thumbnails according to the chroma and brightness blocks of the thumbnails; In the color groups of the thumbnails, the random number determines the color group of a thumbnail; and The representative color of the thumbnail is determined according to the color group of the thumbnail determined by random numbers. The ambient atmosphere adjustment method according to claim 12 or 13, wherein the step of determining the representative color according to the number of pixels in the chroma blocks of the thumbnails includes: obtaining a chromaticity value of a thumbnail from the chromaticity block of the thumbnail corresponding to the color group of the thumbnail; Obtaining a thumbnail's chroma and a thumbnail's lightness value from the chroma-cum-lightness block of the thumbnail corresponding to the color group of the thumbnail; and The representative color of the thumbnail in the RGB color space is determined according to the chroma value of the thumbnail, the chroma value of the thumbnail and the lightness value of the thumbnail. The environment atmosphere adjustment method according to claim 1, wherein the step of determining the representative color of the thumbnail in the foreground of interest of the thumbnail includes: Perform a clustering process on the interest pixels of multiple thumbnails in the interest foreground of the thumbnail to obtain multiple clusters; calculating the number of interesting pixels of the thumbnails in the clustering; The representative color of the thumbnail is determined according to a group cluster having the largest number of the pixels. The environmental atmosphere adjustment method as claimed in claim 15, wherein the grouping and clustering process comprises a K-Mean algorithm or a K-Medoid algorithm. The method for adjusting environmental atmosphere according to claim 2, wherein before the step of making the display play the video frame, further comprising: Recording an ambient sound in the environment in which the video frame is scheduled to be played; and The brightness of the light source is determined according to the volume of the ambient sound. The ambient atmosphere adjustment method according to claim 17, wherein when the volume of the ambient sound is larger, the brightness of the light source is brighter, and when the volume of the ambient sound is smaller, the brightness of the light source is darker. The environmental atmosphere adjustment method according to claim 1, comprising: Determine whether the first lightness of the representative color of the thumbnail is greater than a lightness threshold. The environmental atmosphere adjustment method according to claim 19, comprising: If so, it is determined whether a color difference between the representative color of the thumbnail and the representative color of a previous thumbnail in an RGB color space or an HSV color space is greater than a different color threshold. The environmental atmosphere adjustment method according to claim 19, comprising: If so, it is determined whether the difference in brightness between the representative color of the thumbnail and the representative color of a previous thumbnail is greater than a different color threshold. The environmental atmosphere adjustment method according to claim 20 or 21, comprising: If so, change the color of the light source according to the representative color of the thumbnail. The environmental atmosphere adjustment method according to claim 19, 20 or 21, comprising: If not, the color of the light source is maintained. The environmental atmosphere adjustment method according to claim 19, further comprising: Increase the sampling distance between the image frame and a previous image frame. An ambient atmosphere adjustment system, comprising: a display for playing an image; a light source for emitting light of multiple colors; and a processor, connected to the display and the light source, and configured to: performing a thumbnail processing on the image frame to obtain a thumbnail image frame; Distinguish the interest foreground of a thumbnail according to the brightness of the pixels of the thumbnails in the image frame of the thumbnail; a representative color of a thumbnail that determines the foreground interest of the thumbnail; provide the representative color of the thumbnail to the light source; causing the light source to emit light in the color representative of the thumbnail; and Instruct the display to play the video frame.

Images