JP2006332882A - Moving picture coding apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving picture coding apparatus for achieving a video image with high image quality. <P>SOLUTION: A viewer 100 views a video image displayed on an image display section 101 in the moving picture coding apparatus. A sight line detection section 106 detects a sight line 202 of the viewer 100 viewing the video image to provide an output of view line information 203 being a set of sight line position data. A sight line information analysis section 107 determines a target region by analyzing the sight line information 203 output from the sight line detection section 106 on the basis of coded data 207 output from a video coding section 105 and outputs sight line analysis data 204. A coded parameter setting section 103 inputs the sight line analysis data 204 and the coded data 207 to set and output a coded parameter 205. A video coding section 105 codes a division signal 206 on the basis of the coded parameter 205 to output the coded data 207. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a moving image encoding apparatus for compressing and encoding a moving image.

  As a conventional video encoding device, a video communication system capable of viewing received video that emphasizes the quality of a region of interest in real time by providing a line-of-sight input device, a region of interest detection device, and a video encoding control device Is provided (for example, refer to Patent Document 1).

JP 7-135651 A

Since the conventional video encoding device is configured as described above, when detecting the attention area, the attention area is detected from the line-of-sight position data at a certain time, so when the same video continues, When the line of sight moves frequently, there is a problem that the image looks distorted.
Further, the conventional image encoding device is intended for one person, and there is a problem that it is impossible to use the line-of-sight information of a plurality of persons.

The present invention has been made to solve the above-described problems. By analyzing the line-of-sight position data within a certain period and detecting the attention area in consideration of the line-of-sight information in the time direction, the image quality can be improved. An object of the present invention is to obtain a moving picture coding apparatus that realizes a simple video.
Another object of the present invention is to improve the image quality of a characteristic image by using image feature information (color detection, person detection, character detection, etc.).
It is another object of the present invention to realize a high-quality video by detecting a region of interest using gaze information of a plurality of people.

  The moving image encoding apparatus according to the present invention includes a blocking unit that outputs a divided signal obtained by dividing a video input signal into blocks composed of a plurality of pixels, and at least one for observing the video in synchronization with the input signal. A line-of-sight detection unit that outputs line-of-sight information that detects the line of sight of more than one observer, a line-of-sight information analysis unit that outputs line-of-sight analysis information obtained by analyzing the line-of-sight information, and sets encoding parameters based on the line-of-sight analysis information An encoding parameter setting unit that performs encoding, and a video encoding unit that encodes the divided signal based on the encoding parameter.

  According to the present invention, by analyzing the line-of-sight information within a certain period, it is possible to remove the line-of-sight information when the line of sight is temporarily disturbed from the attention area, and target the area where the line of sight is sufficiently concentrated. A large amount of information can be allocated. Thereby, the image quality of the region where the line of sight is poured can be improved.

Embodiment 1 FIG.
Embodiment 1 of the present invention will be described below. FIG. 1 is a block diagram showing a configuration of a moving picture coding apparatus according to Embodiment 1 of the present invention. In FIG. 1, the moving image encoding apparatus includes an image display unit 101, a line-of-sight detection unit 106, a line-of-sight information analysis unit 107, an encoding parameter setting unit 103, a blocking unit 104, and a video encoding unit 105. It has.

  Next, the operation will be described. The image display unit 101 displays an image based on the input signal 201. Here, it is assumed that the input signal 201 is an input image signal or a prediction error signal and has been decoded. An observer 100 observes an image displayed on the image display unit 101. The line-of-sight detection unit 106 detects the line of sight 202 observed by the observer 100 and outputs line-of-sight information 203 that is a set of line-of-sight position data. The input signal 201 is also input to the line-of-sight detection unit 106 in order to synchronize with the input signal 201 when the line-of-sight detection unit 106 detects the line of sight of the observer 100.

  In parallel with this, the blocking unit 104 divides the input signal 201 into blocks composed of a plurality of pixels and outputs a divided signal 206.

  The line-of-sight information analysis unit 107 analyzes the line-of-sight information 203 output from the line-of-sight detection unit 106 based on the encoded data 207 output from the video encoding unit 105 to determine a region of interest, and the line-of-sight analysis data 204 Is output. Here, the encoded data 207 used for the determination is encoded data of a frame before the present in terms of time (hereinafter referred to as pre-encoded data).

  Hereinafter, the operation of the line-of-sight information analysis unit 107 will be described in detail. FIG. 2 is a diagram showing line-of-sight position data between times T1 and T3 in the first embodiment. 2A shows line-of-sight position data between times T1 and T2, and FIG. 2B shows line-of-sight position data between times T2 and T3. In FIG. 2, when analyzing the distribution of the line-of-sight position data, the line-of-sight information analysis unit 107 divides the screen into 64 as an example. A black circle indicates the position of the line of sight 202 of the observer 100, that is, line-of-sight position data, and shows a state in which the line of sight 202 spreads over time.

  FIG. 3 is a flowchart illustrating a processing flow in which the line-of-sight information analysis unit 107 in FIG. The line-of-sight information analysis unit 107 accumulates the line-of-sight information 203 from the line-of-sight detection unit 106 for a certain period, and analyzes the line-of-sight information 203 for the same period.

  First, in step ST1, the line-of-sight position data in the target block among the blocks divided in FIG. 2 is counted. Then, an accumulated value of the line-of-sight position data in the block (hereinafter, an accumulated value in the block) is calculated.

  Next, in step ST2, it is determined whether or not the intra-block accumulated value is greater than one. If the in-block accumulated value of the target block is greater than 1, the process proceeds to ST3, and if it is 1 or less, the process proceeds to step ST5.

  When the process proceeds to step ST3, “ratio = accumulated value in block / total number of line-of-sight position data” is calculated in step ST3. Then, the ratio of the accumulated value in the block to the total number of line-of-sight position data per picture (hereinafter referred to as the ratio in the block) is obtained.

  Next, the in-block ratio is compared with an arbitrary value X in step ST4. When the in-block ratio is larger than X, the target block is determined as the first attention area. On the other hand, if the in-block ratio is equal to or less than X, the target block is determined as the second region of interest, and the process proceeds to step ST7. Step ST7 will be described later.

  When the process proceeds from step ST2 to step ST5, it is determined whether or not the accumulated value in the block is equal to 1. If the accumulated value in the block is equal to 1, the process proceeds to step ST6. On the other hand, when the intra-block accumulated value is not equal to 1 (that is, when the intra-block accumulated value is 0), the target block is determined as a non-attention area.

  When the process proceeds to step ST6, it is determined whether the accumulated value of the line-of-sight position data in the block adjacent to the target block (hereinafter, the accumulated value of the adjacent block) is zero. When the accumulated value of the adjacent block is 0, the target block is determined as a non-target area. On the other hand, if the accumulated value of the adjacent block is not 0, the target block is determined as the second region of interest, and the process proceeds to step ST7.

  Next, in step ST7, it is determined whether or not the adjacent block is a non-target area. If the adjacent block is a non-attention area, the target block remains as the second attention area. On the other hand, if the adjacent block is not a non-attention area, that is, if it is an attention area, the target block is changed to the first attention area.

  In FIG. 3, the attention areas are two types of the first attention area and the second attention area, but the attention areas may be classified into more attention areas. Further, although the attention area is determined based on the accumulated value of the line-of-sight position data, the attention area may be determined using a value obtained by multiplying the line-of-sight position data by the gaze time as an accumulated value.

  The image quality of the attention area is improved by increasing the target information amount of the block including the attention area determined by the line-of-sight information analysis section 107 among the blocks divided by the blocking section 104. In FIG. 2, since the attention area is expanded, the number of blocks that increase the target information amount increases with time. Then, the area other than the attention area (non-attention area) reduces the target information amount so that the total target information amount in one picture does not change.

  FIG. 4 is a diagram illustrating a setting change example of the target information amount in the first embodiment. In FIG. 4, one picture is divided into 64 as an example. FIG. 4A shows an initial value of the target information amount. The target information amount of each block obtained by dividing the target information amount for one picture (6400 bits) by the total number of blocks (64) is the same value (100 bits). Become.

  FIG. 4B is a diagram in which a region of interest is designated from the line-of-sight position data of FIG. A hatched block in FIG. 4B is a region of interest. The blocks B1, B2, B3, and B4 in FIG. 2A are determined as the first region of interest because the line-of-sight position data is concentrated. The B5, B6, and B7 blocks have only one line-of-sight position data, and the line-of-sight position data also does not exist in adjacent blocks, so that they are determined as non-attention areas. Since the periphery of the attention area is easily visually recognized, the second attention area that does not reduce the target information amount is set. In FIG. 4B, as an example, the target information amount of the first region of interest (hatched block) is 2.20 times, and the target information amount of the second region of interest (horizontal line block) is 1.00 times (initial setting). The target information amount of the non-attention area (blank block) is multiplied by 0.90. The total target information amount of one picture is the same in both FIG. 4 (a) and FIG. 4 (b).

  In FIG. 1 again, the encoding parameter setting unit 103 inputs the line-of-sight analysis data 204 and the encoded data 207, sets the encoding parameter 205, and outputs it. The encoded data 207 input here is pre-encoded data as in the line-of-sight information analysis unit 107.

  The video encoding unit 105 encodes the divided signal 206 based on the encoding parameter 205 and outputs encoded data 207. The encoded data 207 is used by the line-of-sight information analysis unit 107 to determine a region of interest when encoding subsequent frames.

  As described above, according to the first embodiment, by analyzing the line-of-sight information 203 within a certain period, the line-of-sight information 203 when the line of sight is temporarily disturbed can be removed from the region of interest. A large amount of target information can be assigned to a sufficiently concentrated area. As a result, the image quality of the region where the line of sight is poured can be improved.

Embodiment 2. FIG.
The second embodiment of the present invention will be described below. FIG. 5 is a block diagram showing the configuration of the moving picture coding apparatus according to Embodiment 2 of the present invention. The moving picture coding apparatus according to Embodiment 2 has a configuration in which an image feature extraction unit 102 is added to the moving picture coding apparatus (FIG. 1) according to Embodiment 1. Since other configurations are the same as those of the first embodiment, description thereof is omitted.

  Next, the operation will be described. The image feature extraction unit 102 receives the input signal 201, extracts features included in the image, and outputs an image feature signal 208. The encoding parameter setting unit 103 sets an encoding parameter from the image feature signal 208, the line-of-sight analysis data 204, and the pre-encoded data 207. The blocking unit 104 divides the input signal 201 into blocks composed of a plurality of pixels and outputs a divided signal 206. The video encoding unit 105 encodes the divided signal 206 based on the encoding parameter 205 and outputs encoded data 207. Since other operations are the same as those in the first embodiment, description thereof is omitted.

  Hereinafter, the operation of the image feature extraction unit 102 will be described in detail. FIG. 6 is a diagram showing line-of-sight position data within a certain period in the second embodiment. As an example, the screen is divided into 64 parts. The line-of-sight position data is concentrated in an ellipse near the center of the screen, but the line-of-sight position data is also scattered outside the ellipse. Here, when the image feature extraction unit 102 detects that the person and the character are reflected in the same screen, the line-of-sight position data scattered outside the ellipse is an area in which the person and the character are reflected (hereinafter, referred to as “line”). It can be seen that it is included in the feature region.

  The image quality of the attention area is improved by increasing the target information amount of the block including the attention area determined by the line-of-sight information analysis section 107 among the blocks divided by the blocking section 104. Also, when the line of sight is directed to the feature area detected by the image feature extraction unit 102, the image quality is improved by increasing the target information amount of the block. The target information amount is reduced in regions other than the attention region and the feature region, so that the total target information amount in one picture does not change.

  FIG. 7 is a diagram illustrating a setting change example of the target information amount in the second embodiment. As an example, the screen is divided into 64 parts. FIG. 7A shows an initial value of the target information amount, which is obtained by dividing the target information amount for one picture by the total number of blocks, and each block has the same value. FIG. 7B is a diagram in which a region of interest and a feature amount region are designated from the line-of-sight position data of FIG. In FIG. 7B, the target information amount of the attention area (shaded block) is 2.00 times, the target information amount of the feature area (wavy line block) is 1.60 times, and the periphery of the attention area and the feature area (horizontal line block). The target information amount of the non-target area (blank block) is multiplied by 0.80 times. The total target information amount of one picture is the same in both FIG. 7 (a) and FIG. 7 (b).

  As described above, according to the second embodiment, by analyzing the line-of-sight information 203 within a certain period and further considering the characteristics of the image, the area where the line of sight is sufficiently concentrated and the line of sight are concentrated. Even if it is not, a large amount of target information can be assigned to a region having an image characteristic. Thereby, the image quality of the region where the line of sight is poured and the characteristic region can be improved.

Embodiment 3. FIG.
The third embodiment of the present invention will be described below. FIG. 8 is a block diagram showing the configuration of the moving picture coding apparatus according to Embodiment 3 of the present invention. The moving picture coding apparatus according to Embodiment 3 has a configuration in which attention area calculation section 109 and attention area storage section 110 are added to the moving picture coding apparatus (FIG. 5) according to Embodiment 2. Note that the upper and lower stages in FIG. 8 are continuous with the attention area storage unit 110 as a connection point, but the upper and lower stages operate separately. Since other configurations are the same as those of the second embodiment, description thereof is omitted.

  Next, the operation will be described. The input image data 108 is a medium storing video. The image display unit 101 displays a video based on the input signal 201 of the input image data 108. An observer 100 observes an image displayed on the image display unit 101. The line-of-sight detection unit 106 detects the line of sight 202 observed by the observer 100 and outputs line-of-sight information 203. The line-of-sight information analysis unit 107 analyzes the line-of-sight information 203 and outputs line-of-sight analysis data 204. The image feature extraction unit 102 extracts features included in the image and outputs an image feature signal 208.

  The attention area calculation unit 109 calculates an attention area of the image based on the input line-of-sight analysis data 204 and the image feature signal 208 and outputs an attention area specifying signal 209. The attention area storage unit 110 receives the attention area identification signal 209 and stores information on the entire video indicating what kind of line of sight is being poured.

  The encoding parameter setting unit 103 sets and outputs the encoding parameter 205 based on the attention area accumulation data 210 (the entire video information is stored) output from the attention area accumulation section 110. Since the attention area storage unit 110 stores information of the entire video, not only past information that has been encoded but also future information that has not yet been encoded can be used. For example, when it is known that the area where the line of sight concentrates increases, the image quality can be improved by setting a wide area to which a large amount of target information is allocated in advance.

  When it is known that the gaze time of a line of sight for a certain block is long, the image quality can be improved by assigning a large amount of information of the block from the time when the line of sight begins to gaze.

  The blocking unit 104 divides the input signal 201 into blocks composed of a plurality of pixels and outputs a divided signal 206. Here, the input signal 211 is the same signal as the input signal 201. The video encoding unit 105 encodes the divided signal 206 based on the encoding parameter 205 and outputs encoded data 207.

  As described above, according to the third embodiment, since the line-of-sight analysis data 204 and the image feature signal 208 of the entire video are stored in the attention area storage unit 110, only past information is encoded. In addition, future information can be used. This makes it possible to allocate the amount of information in advance to an area where the line of sight will be poured in the future, and to improve image quality.

Embodiment 4 FIG.
The fourth embodiment of the present invention will be described below. FIG. 9 is a block diagram showing the configuration of the moving picture coding apparatus according to Embodiment 4 of the present invention. The moving picture coding apparatus according to Embodiment 4 has a configuration in which learning data calculation section 111 and learning data storage section 112 are added to the moving picture coding apparatus (FIG. 5) according to Embodiment 2. Since other configurations are the same as those of the second embodiment, description thereof is omitted.

  Next, the operation will be described. The image display unit 101 displays an image based on the input signal 201. An observer 100 observes an image displayed on the image display unit 101. The line-of-sight detection unit 106 detects the line of sight 202 observed by the observer 100 and outputs line-of-sight information 203. The line-of-sight information analysis unit 107 analyzes the line-of-sight information 203 based on the pre-encoded data 207 and outputs the line-of-sight analysis data 204. The image feature extraction unit 102 extracts features included in the image and outputs an image feature signal 208.

  The learning data calculation unit 111 calculates a region of interest based on the line-of-sight analysis data 204 and the image feature signal 208 and outputs a learning data signal 212. The learning data accumulation unit 112 accumulates image features that the observer 100 often pays attention as learning data (learning data signal 212). Then, accumulated learning data 213 in which the learning data signal 212 is accumulated is output.

  The encoding parameter setting unit 103 sets and outputs an encoding parameter 205 based on the line-of-sight analysis data 204, the image feature signal 208, the accumulated learning data 213, and the pre-encoded data 207. Since the learning data storage unit 112 stores image features (accumulated learning data 213) that the observer 100 often pays attention to, the current input image features and the stored learning data 213 are compared for observation. If it is an area that the person 100 is interested in, the encoding parameter setting unit 103 sets the encoding parameter 105 so as to increase the target information amount.

  Hereinafter, a setting example of the encoding parameter 205 in the encoding parameter setting unit 105 will be shown. For example, when it is known from the learning data that the observer 100 tends to focus on a red video, when the red color is detected in the input signal 201, the target information amount of the corresponding block is increased. The image quality is improved by setting.

  For example, if the learning data indicates that the observer 100 tends to focus on moving images rather than a stationary image (background), a moving region is detected in the input signal 201. In this case, image quality is improved by setting a large amount of target information for the corresponding block.

  For example, when it is known from the learning data that the line of sight of the viewer 100 tends to move, the amount of information is concentrated in a specific area by suppressing an increase in the amount of target information allocated to the attention area. The image quality is improved by avoiding the problem and distributing the information amount to the entire screen.

  For example, when it is known from the learning data that the line of sight of the viewer 100 tends to concentrate near the center of the screen, the image quality is improved by setting a large amount of target information near the center of the screen.

  In FIG. 9 again, the blocking unit 104 divides the input signal 201 into blocks composed of a plurality of pixels and outputs a divided signal 206. The video encoding unit 105 encodes the divided signal 206 based on the encoding parameter 205 and outputs encoded data 207.

  As described above, according to the fourth embodiment, since the features of the image that the observer 100 tends to focus on are accumulated in the learning data accumulation unit 112 as learning data, the code specialized for the observer 100 is used. Image quality can be improved, and image quality can be improved.

Embodiment 5. FIG.
The fifth embodiment of the present invention will be described below. FIG. 10 is a block diagram showing the configuration of the moving picture coding apparatus according to Embodiment 5 of the present invention. Since the moving picture encoding apparatus according to Embodiment 5 has the same configuration as the moving picture encoding apparatus (FIG. 8) according to Embodiment 3, the description thereof is omitted. However, it is different that there are a plurality of observers 100.

  Next, the operation will be described. Only the differences from the third embodiment will be described. FIG. 11 shows line-of-sight position data of three observers 100 between times T1 and T2 in the fifth embodiment. As an example, the screen is divided into 64 parts. In FIG. 11, black circles, diamonds, and crosses represent line-of-sight position data of the three observers 100. When there are a plurality of observers 100, a specific observer 100 can be weighted when counting line-of-sight position data. That is, when emphasizing the line-of-sight information 203 of a specific observer 100, the line-of-sight position data of the specific observer 100 is multiplied by α (> 1.0) and counted.

  FIG. 12 is a diagram illustrating a setting change example of the target information amount in the fifth embodiment. As an example, the screen is divided into 64 parts. FIG. 12A shows an initial value of the target information amount, which is obtained by dividing the target information amount for one picture by the total number of blocks, and has the same value for each block. FIG. 12B is a diagram in which a region of interest and a feature region are designated from the line-of-sight position data of FIG. In FIG. 12B, the target information amount of the attention area 1 (hatched block) is 2.05 times, the target information amount of the attention area 2 (dashed block) is 1.55 times, and the area around the attention area (horizontal line block) is increased. The target information amount is increased by 1.00 (initially set), and the target information amount of the non-attention area (blank block) is increased by 0.80. The total target information amount of one picture is the same in both FIG. 12 (a) and FIG. 12 (b).

  As described above, according to the fifth embodiment, since the line-of-sight information of the entire images of a plurality of people is accumulated, the influence of the personal characteristics of the observer 100 is reduced and more general line-of-sight information is used. can do. In addition, since line-of-sight information and image feature information are stored, not only past information but also future information can be used for encoding. This makes it possible to allocate the amount of information in advance to an area where the line of sight will be poured in the future, and to improve image quality.

  In the fifth embodiment, a plurality of observers 100 according to the third embodiment are used. However, in another embodiment, a plurality of observers 100 may be used.

  In Embodiments 1, 2, and 5, FIGS. 2, 4, 6, 7, 11, and 12 divide the screen into 64, but the number of divisions is not necessarily limited to 64.

  Needless to say, the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or apparatus.

It is a block diagram which shows the structure of the moving image encoder which concerns on Embodiment 1 of this invention. In Embodiment 1, it is the figure which showed the gaze position data between the times T1-T3. It is a flowchart which shows the processing flow which the gaze information analysis part 107 in FIG. 1 determines an attention area. In Embodiment 1, it is the figure which showed the example of setting change of the target information amount. It is a block diagram which shows the structure of the moving image encoder which concerns on Embodiment 2 of this invention. In Embodiment 2, it is the figure which showed the gaze position data within a certain period. In Embodiment 2, it is the figure which showed the example of setting change of the target information amount. It is a block diagram which shows the structure of the moving image encoder which concerns on Embodiment 3 of this invention. It is a block diagram which shows the structure of the moving image encoder which concerns on Embodiment 4 of this invention. It is a block diagram which shows the structure of the moving image encoder which concerns on Embodiment 5 of this invention. In Embodiment 5, the line-of-sight position data of three observers 100 between times T1 and T2 are shown. In Embodiment 5, it is the figure which showed the example of setting change of the target information amount.

Explanation of symbols

100 observer, 101 image display unit, 102 image feature extraction unit, 103 coding parameter setting unit, 104 blocking unit, 105 video coding unit, 106 gaze detection unit, 107 gaze information analysis unit, 108 input image data, 109 Region-of-interest calculation unit 110 Region-of-interest storage unit 111 Learning data calculation unit 112 Learning data storage unit 201 Input signal 202 Line of sight 203 Line-of-sight information 204 Line-of-sight analysis data 205 Coding parameter 206 Divided signal 207 Code Data, 208 image feature signal, 209 attention area specifying signal, 210 attention area accumulation data, 211 input signal, 212 learning data signal, 213 accumulation learning data.

Claims (7)

A blocking unit that outputs a divided signal obtained by dividing a video input signal into blocks composed of a plurality of pixels;
A line-of-sight detection unit that outputs line-of-sight information obtained by detecting the line of sight of at least one observer who observes the video in synchronization with the input signal;
A line-of-sight information analysis unit that outputs line-of-sight analysis information obtained by analyzing the line-of-sight information;
An encoding parameter setting unit that sets an encoding parameter based on the line-of-sight analysis information;
A video encoding apparatus comprising: a video encoding unit that encodes the divided signal based on the encoding parameter.   The moving image encoding apparatus according to claim 1, wherein the line-of-sight information analysis unit analyzes a distribution of the line-of-sight information in a time axis direction or a space axis direction.   The moving image encoding apparatus according to claim 1, wherein the line-of-sight information analysis unit performs weighting according to the observer when analyzing the line-of-sight information of a plurality of the observers. An image feature extraction unit for outputting an image feature signal obtained by extracting an image feature from the input signal;
4. The moving image according to claim 1, wherein the encoding parameter setting unit sets the encoding parameter based on the line-of-sight analysis information and the image feature signal. 5. Image encoding device. A region-of-interest calculator that outputs a region-of-interest specifying signal that calculates the region of interest of the image from the line-of-sight analysis data and the image feature signal;
A region-of-interest storage unit that outputs region-of-interest storage data that stores the region-of-interest specifying signal for the entire video;
5. The moving image encoding apparatus according to claim 4, wherein the encoding parameter setting unit sets the encoding parameter based on the attention area accumulation data. A learning data calculation unit that outputs a learning data signal obtained by learning the line-of-sight information based on the line-of-sight analysis data and the image feature signal;
A learning data accumulation unit that outputs accumulated learning data in which the learning data signal is accumulated;
5. The moving image encoding apparatus according to claim 4, wherein the encoding parameter setting unit sets the encoding parameter based on the line-of-sight analysis information, the image feature signal, and the accumulated learning data. The moving image encoding apparatus according to claim 6, wherein the learning data calculation unit learns a tendency of the observer's line of sight and outputs the learning data signal.

Images