JPH10271515A - Noticed area tracing method, noticed area tracing device using the method and image coding method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To surely detect a noticed area by avoiding the hardware from being enlarged to prevent the processing time from being extended. SOLUTION: Inverse detection (S1) is conducted, where a corresponding position of an image block arranged in a current frame is searched in a preceding frame. For example, the result of block matching by the motion picture experts group(MPEG) is used as it is. Then a forward detection is conducted, where a corresponding position of the noticed area taken in a preceding frame is searched in a current frame (S2). Thus, the noticed area is set also to the current frame. Then a much more coding quantity is given to the noticed area and the frame image is coded (S3).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an attention area tracking method, an attention area tracking apparatus, and an image encoding method.
More particularly, the present invention relates to a method of tracking a region of interest using detection of motion of a unit region included in a frame image, a region of interest tracking device using the method, and an image encoding method.

[0002]

2. Description of the Related Art With the dramatic improvement in hardware performance and software technology of various network technologies and information devices, applications for transmitting and utilizing a large amount of image data are expanding. With the progress of image processing technology, cameras, personal computers, video equipment, televisions and other image-related equipment have started to have organic connections, and have been combined with network-related infrastructure such as the Internet to create new image cultures and business forms. Has begun to provide an environment for education and business. Under these circumstances, there is a great demand for transmitting or providing a higher quality moving image at a higher speed and with a smaller data amount.

At present, various techniques for encoding moving images at a high data compression rate have been put to practical use. For example, when considering a videophone, there is a known idea of compressing data by increasing the encoding accuracy of a region of interest centered on the face of a speaker and reducing the encoding accuracy of a region of non-interest such as the background as much as possible. I have. Various image processing techniques such as an image recognition technique are used to detect a region of interest from a frame image.

[0004]

However, the calculation time required to detect a region of interest from a frame image is generally long. In particular, when high detection accuracy is required, the hardware required for calculation increases, which hinders cost reduction in commercialization and real-time processing of moving images. There is a demand for a technique for avoiding enlargement of hardware and accurately detecting a region of interest without increasing processing time.

Japanese Patent Laid-Open Publication No. Hei 8-18192 discloses a moving image encoding apparatus and an area extracting apparatus which extract a specific area and a moving area from an input image and allocate a relatively large amount of code to the specific area. Have been. According to the publication, when a videophone is considered, the specific area can be understood as the face of the speaker or the vicinity of the center thereof, and the motion area can be understood as the surrounding moving area. In this method, a specific area and a motion area are detected when performing motion compensation prediction. However, in the case of this device, since a two-step process of obtaining a specific area and then obtaining a motion area is performed, there is room for improvement in processing load and required hardware size.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and an attention area tracking method that hardly requires additional hardware to detect an attention area and hardly increases the processing time, and a method thereof. An object of the present invention is to provide an attention area tracking device and an image encoding method using the method.

[0007]

One method of tracking a region of interest according to the present invention is based on the premise that a frame image is divided into unit regions such as blocks and areas, and movement between frames is tracked for each unit region. And In this case, the present invention is performed independently of the unit area (that is, without being constrained by the unit area, not conscious of the unit area, or a boundary line not related to the boundary line of the unit area). Rectangle), a circle,
The motion of the attention area having an arbitrary shape such as the outer shape of the object is estimated using the tracking result of the movement of the unit area having a predetermined positional relationship with the attention area. The tracking of the motion between frames performed for each unit area may be originally performed independently of the tracking of the attention area as part of the motion compensation prediction. The “predetermined positional relationship” may be, for example, a case where the attention area includes the unit area, a case where the attention area includes a part of the unit area, or a case where the unit area exists near the attention area.

According to the present invention, when the motion of the unit area is detected for the purpose of motion compensation prediction, the detection result is effectively used for detecting the motion of the attention area. Therefore, there is no need to prepare hardware dedicated to tracking the movement of the attention area, and the processing time that is increased by tracking the attention area is very small.

Another method of tracking a region of interest according to the present invention includes a backward detecting step and a forward detecting step. The reverse direction detection step includes:
The position of a region (also simply referred to as a corresponding position) corresponding to a unit region provided in the current frame (including the entire frame based thereon) is detected in the previous frame (including the entire frame based thereon). On the other hand, in the forward direction detecting step, the position of the area corresponding to the attention area provided in the previous frame, that is, the corresponding position is detected in the current frame by using the result of the backward direction detecting step. This method also has the same effect as described above.

[0010] One image encoding method according to the present invention comprises:
It is assumed that a frame image is divided into unit areas, and movement between frames is tracked for each unit area. At this time, the present invention uses the tracking result of the movement of the attention area of an arbitrary shape defined in the frame image independently of the unit area by using the movement result of the unit area having a predetermined positional relationship with the attention area. The estimated region of interest in the estimated and frame image is encoded with processing different from that of the other regions. According to the present invention, for example, a data compression ratio can be further increased by adding very little hardware to an existing apparatus that performs motion compensation prediction and encodes an image. The processing content may be changed so that the encoding accuracy of the region of interest is increased, that is, when the region of interest is decoded, the image quality of the region is higher than that of other regions. The “different processing contents” include a case where a different coding method is used and a case where a parameter is changed in the same coding method. If the quantization is performed before and after the encoding, the encoding process is also referred to as including the quantization.
Therefore, even when the number of quantization steps is changed, the contents of the encoding process are considered to be different.

Another image encoding method according to the present invention includes an encoding step in addition to the above-described backward detection step and forward detection step. In the encoding step, the attention area of the frame image is encoded with processing different from that of other areas. Therefore, as described above, a high data compression ratio can be realized while, for example, increasing the encoding accuracy of the attention area.

On the other hand, the attention area tracking apparatus of the present invention uses means for determining the position (corresponding position) of the area corresponding to the unit area provided in the current frame in the previous frame, and utilizing the determination result. Means for setting the position (corresponding position) of the region corresponding to the region of interest provided in the previous frame in the current frame. According to this configuration, for example, when information on the motion of the unit area is obtained by the motion compensation prediction, the motion of the attention area can be tracked using the information. Therefore, the addition of hardware and the increase in processing time are very small. Note that an encoding unit that encodes the set attention area with processing different from that of the other areas may be added to the attention area tracking apparatus.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described with reference to the drawings as appropriate. FIG. 1 is a flowchart showing an outline of a processing procedure of the attention area tracking method and the image encoding method according to the embodiment. As shown in the figure, the processing includes backward detection (S1), forward detection (S2), and encoding (S
Continue with 3). Among them, the backward direction detection and the forward direction detection correspond to the attention area tracking method, and if the encoding is included, the method becomes the image encoding method.

In the backward direction detection (S1), the position of an area corresponding to the unit area provided in the current frame (hereinafter, also simply referred to as "unit area corresponding position") is detected in the previous frame. That is, a region where the corresponding region in the previous frame of the unit region correctly aligned with the current frame is likely to be located is detected as a candidate region, and for each unit region, motion information indicating the displacement of itself and the candidate region is displayed. Generated. This motion information is, for example, a motion vector.

In the present embodiment, motion compensation prediction employed in MPEG or the like is used to detect a corresponding position in a unit area. In MPEG, blocks which are unit areas are arranged in a matrix in the current frame, and the corresponding positions of these blocks are searched for in the previous frame. Therefore, the compatibility with the present embodiment is high. However, the detection of the corresponding position in the unit area does not necessarily need to use motion compensation prediction, but may be performed using image analysis such as edge detection or optical flow. In that case, the detection of S1 need not be in the reverse direction.

Subsequently, forward direction detection (S2) is performed.
In S2, by using the result of S1, the position of the region corresponding to the region of interest provided in the previous frame (hereinafter, referred to as the position of the region)
Is simply detected in the current frame. The attention area is preferably larger than the unit area. In the present embodiment, the unit region and the attention region have the same shape and the same size in each frame, but differ when a part of the region extends outside the frame.

In S2, of the candidate regions taken in the previous frame, those included in the region of interest in the previous frame are:
First, it is selected as an important area. Next, the attention area is tracked by comprehensively considering the motion information on the important area. As an example, the motion vector of the attention area may be obtained by averaging the motion vectors of the important area. As a result, the attention area of the current frame is determined based on the attention area set in the previous frame, and the attention area can be tracked sequentially.

Thereafter, encoding (S3) is performed. Since the region of interest in the current frame is determined in S2, it is possible to perform encoding such that the encoding accuracy of the region of interest is increased and the encoding accuracy of the other regions is kept low. By appropriately adjusting the content of the encoding process, the data compression ratio can be increased while maintaining the image quality, especially the subjective image quality, at a sufficiently high level.

FIG. 2 is a configuration diagram of an image encoding apparatus according to the embodiment. In the figure, the attention area tracking unit 13 and the input to the attention area tracking unit 13 and the code amount control unit 14 responding to the output of the attention area tracking unit 13 have a new configuration, and the other components use a known MPEG encoder. I have.

As shown in FIG. 1, the apparatus comprises an input frame memory unit 1 for sequentially inputting frame images, a subtracter 2 for subtracting an output 21 of the motion compensation prediction unit 12 from the frame image which is an output 20, and an output of the subtracter 2. 22, an orthogonal transform unit 3 for applying an orthogonal transform such as a discrete cosine transform (DCT) thereto, and an output 23 of the orthogonal transform unit 3 and a quantizing unit 4 for quantizing coefficients of each spatial frequency component. The output 24 of the quantization unit 4 is input, the inverse quantization unit 7 for performing inverse quantization on this, the output 27 of the inverse quantization unit 7 is input, and the inverse orthogonal transformation unit 8 for performing inverse orthogonal transformation on this, An adder 9 which receives the output 28 of the orthogonal transform unit 8 and adds the output 21 of the motion compensation prediction unit 12 thereto, the reference frame memory unit 10 which receives the output 29 of the adder 9 and stores it, a reference frame The output 30 of the memory unit 10 and The input frame memory section 1 outputs 2
0, and performs a block matching to derive a motion vector 15 for each block. The motion detection unit 11 predicts the pixel value of each block of the current frame based on the output 30 of the reference frame memory unit 10 and the motion vector 15. The motion compensation prediction unit 12 and the output 24 of the quantization unit 4 are input, and the encoding unit 5 for entropy encoding each coefficient of the quantized spatial frequency and the output 25 of the encoding unit 5 are input. The buffer memory unit 6 for temporarily storing the timing, the motion vector 15 is input, the attention area tracking unit 13 for tracking the attention area, and the first encoding output from the buffer memory unit 6 based on the buffer availability. An accuracy instruction 26 and a second encoding accuracy instruction 16 output from the attention area tracking unit 13 to indicate the position of the attention area are input, and these two instructions are used. With code amount control unit 14 for controlling the coding accuracy me. The output 31 of the code amount control unit 14 is provided to the quantization unit 4.

The quantizer 4 outputs the output 3 of the code amount controller 14
The quantization step (quantization width) is changed so that the code amount or the coding precision changes according to 1. That is, the quantization step is set finely when the coding accuracy is to be increased for the attention area, and coarsely when the coding area is not the attention area. Separately from this, when the free space of the buffer in the buffer memory unit 6 decreases, the first encoding accuracy instruction 26 instructs to reduce the number of quantization levels.

FIG. 3 is an internal configuration diagram of the attention area tracking unit 13. The attention area tracking unit 13 receives the motion vector 15 and determines whether or not the corresponding position of each block of the current frame in the previous frame falls within the attention area set in the previous frame. According to the instructions of the initial setting unit 131, the block position determining unit 130, and the initial setting unit 131, which initially set the attention area for the first frame to be input in the processing of the above, the corresponding position in the previous frame is set to the previous frame. The attention area setting unit 132 calculates the motion vector of the attention area by comprehensively considering the motion vector of the block that falls within the attention area, and sets the attention area in the current frame based on the motion vector. Position holding for temporarily storing the position information of the attention area output in the form of the second encoding accuracy instruction 16 from the area tracking unit 13 Including the 133. The position information of the attention area is updated every frame, and the block position determination unit 13
0 is referred to when processing the next frame.

The tracking of the attention area and the encoding of the image by the above configuration will be described. First, it is assumed that a certain frame _Fn is input to the input frame memory unit 1. The frame _Fn is output to the subtractor 2. The output 21 of the motion compensation prediction unit 12 is input to the subtractor 2. The output 21 is, frames that are generated by predicting the current frame F _n frame F _n-1 inputted to the previous Based, namely MPEG
Is called a reference frame.
Only the difference between the actually input frame _Fn and the reference frame is output to the orthogonal transformer 3 as the output 22 of the subtractor 2.

The orthogonal transform unit 3 orthogonally transforms the difference,
Expand to spatial frequency components. The result of the expansion is orthogonal transform unit 3.
Is output to the quantization unit 4 as an output 23. Quantization unit 4
Quantizes each coefficient of the spatial frequency component. At this time, the quantization unit 4 determines the number of quantization steps according to the instruction of the code amount control unit 14.

The output 24 of the quantization unit 4 is supplied to the encoding unit 5, where the quantized coefficients are encoded. The output 25 of the encoding unit 5 is supplied to the buffer memory unit 6, where the timing is adjusted, and then output to a data storage medium or a decoder (not shown).

The output 24 of the quantization unit 4 is also supplied to the inverse quantization unit 7, where the inverse quantization is performed. The output 27 of the inverse quantization unit 7 is input to the inverse orthogonal transform unit 8, where the inverse orthogonal transform is performed. At this stage, an image corresponding to the difference is reproduced. The output 28 of the inverse orthogonal transform unit 8 is provided to the adder 9. The adder 9 adds the output 28 of the inverse orthogonal transform unit 8 and the output 21 of the motion compensation prediction unit 12 to generate a reference frame. The reference frame is input to the reference frame memory unit 10 as an output 29 of the adder 9 and stored.

On the other hand, the next frame F _{n + 1} is input to the input frame memory unit 1 and the processing of this frame is started. A reference frame generated based on the frame F _n ,
That is, the frame referred to in the processing of the frame F _{n + 1} is output as the output 30 of the reference frame memory unit 10 at an appropriate timing by the motion compensation prediction unit 12 and the motion detection unit 11.
Given to. The motion detecting unit 11 outputs a frame F which is an output 20 of the reference frame and the input frame memory unit 1.
Perform block matching between _{n + 1} . In the case of MPEG, this block matching is such that for each of the blocks aligned in the later frame, frame F _{n + 1} , the corresponding position is shifted from the previous frame, frame F _n , more precisely from frame F _n. Go to the generated reference frame. Therefore, this processing is backward detection. If the corresponding position is found for each block, a motion vector 15 is determined, and this is output to the motion compensation prediction unit 12 and the attention area tracking unit 13.

The motion compensation prediction unit 12 moves each block in the reference frame according to the motion vector, and generates an image closest to the frame F _{n + 1} . Thus, the motion compensated prediction is completed, and the output 2 of the motion compensated prediction unit 12 is output.
1 is output to the subtractor 2. In order to compensate for the motion, the subtracter 2 generally requires a smaller amount of difference data and a smaller code amount. Thereafter, the same processing is repeated to sequentially encode the frame images.

The block position determination unit 130 of the attention area tracking unit 13 determines whether or not the corresponding position of each block of the current frame in the previous frame falls within the attention area set in the previous frame based on the motion vector 15. judge. In the present embodiment, when the center of the corresponding position is included in the attention area, it is regarded as “entering the attention area”, and such a block is referred to as an “important block”.

The attention area setting unit 132 calculates the average of the motion vector of the important block, and obtains the motion of the attention area from the average vector. Therefore, the position of the attention area of the current frame is determined by adding the displacement indicated by the average vector from the position of the attention area of the previous frame. According to this position, the attention area of the current frame is set. The set position information of the attention area is stored in the position holding unit 133 and used for processing of the next frame. This is forward detection because the region of interest is detected as the current frame from the previous frame and the next frame from the current frame. The attention area can be tracked by repeating the determination of the important block and the calculation of the average vector.

It is necessary to set a region of interest for the first frame of a moving image by any method.
This is handled by the initial setting unit 131. The initial setting unit 131 may have a user interface function of receiving an input of a region of interest by a user, or may have an automatic search function of searching for a region of interest by a known method. Further, in the case of a videophone or the like, since the attention area often exists near the center of the screen, the initial setting unit 131 may fixedly set the first attention area at the center of the frame image.

Details of the backward direction detection and the forward direction detection by the present apparatus will be described with reference to FIGS. FIG. 4 is a flowchart of the processing procedure by this apparatus, and FIG.
6 shows the six blocks A to F aligned in _n , FIG. 6 shows the corresponding positions of these blocks in the previous frame F _n-1 and the attention area 40 set in that frame, and FIG. 7 shows the current frame F _n FIG. 9 is a diagram showing a focused attention area 41 together with a attention area 40 of a previous frame F _n-1 . FIG.
Is a detailed example of one of the flowcharts in FIG. FIG.
5 correspond to the blocks A to F in FIG. 5, respectively.

As shown in FIG. 4, it is first determined whether or not the frame to be processed is the first frame (S10). If the frame is the first frame, a region of interest is set in the frame by the initial setting unit 131 (S11). Otherwise, the reverse direction detection (S
Proceed to 12).

For backward detection, first, the current frame F _n
, A reference frame is generated based on the previous frame F _n-1 (S120). The reference frame is generated along the path from the input frame memory unit 1 to the adder 9 in FIG. Subsequently, the motion detection unit 11 performs block matching,
The motion vector 15 of each block is calculated (S12
1). How the block is aligned with the current frame F _n during reverse direction detection is shown in FIG.

Next, the motion vector 15 is given to the attention area tracking unit 13, and the forward direction detection is started (S1).
3). The block position determining unit 130 selects an important block based on the motion vector 15 (S130). In the case of FIG. 6, among the blocks A 'to F', four blocks B ', C', E ', and F' are included in the attention area 40 set in the previous frame _Fn-1. Therefore, these (or blocks B, C, E, and F corresponding to them) are important blocks. Now, 'position vector, namely a motion vector (U _B, V _B), similarly blocked C for block C' Block B to Block B motion vectors of (U _C, V _C), block E 'for block E of the motion vector (U _E, V _E), the motion vector of the block F 'for the block F (U _F,
V _F ), the above average vector (U _AVE , V _AVE )
Is ((U _B + U _C + U _E + U _F ) / 4, (V _B + V _C + V _E +
V _F ) / 4). Attention area setting unit 132, the position vector of the target region 41 in the current frame F _n for the attention area 40 of the previous frame F _n-1, also contemplates _{direction, (- (U B + U} C + U E + U F) / 4 ,-(V _B + V _C + V _E
+ V _F ) / 4). The vector basis, by moving the frame F _n-1 of the target area 40 before as in Fig. 7 sets a region of interest 41 in the current frame F _n (S131), the attention area tracking unit 13 the position information The output is transmitted to the code amount control unit 14 as the second encoding precision instruction 16.

Thereafter, encoding of the frame image is performed (S14). The code amount control unit 14 controls the quantization unit 4 according to the position information of the region of interest 41 such that the code amount is relatively large with respect to the region of interest 41 and the code amount is relatively small with respect to the other regions. I do.

Experimental Results In order to test the usefulness of the present embodiment, an experiment was performed on the image encoding apparatus shown in FIG. The frame image used in the experiment was an MPEG4 test image "foreman"
(CIF 352 × 288, 100 frames). QCIF size (176
× 144). As an example of the present embodiment, the region of interest is the quantization parameter QP = 4 of the DCT coefficient.
, And the other areas were coded with QP = 10. The smaller the value of QP, the higher the encoding accuracy.

1. In the case of the condition that the image quality of the attention area is the same In the conventional general method, the entire frame image is encoded with the same encoding precision (here, QP = 4). In this embodiment, the total code amount can be reduced by 39% as compared to that case.

2. When the Entire Code Amount is Constant When Encoding the Entire Frame Image with Constant Encoding Accuracy, QP = 6 makes the overall code amount almost equal to that of the present embodiment. Compared to that case, the image quality of the attention area is 1.8d.
bImproved.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating an outline of a processing procedure of an attention area tracking method and an image encoding method according to an embodiment.

FIG. 2 is a configuration diagram of an image encoding device according to an embodiment.

FIG. 3 is an internal configuration diagram of an attention area tracking unit 13 in FIG. 2;

FIG. 4 is a flowchart of a processing procedure by the device according to the embodiment.

FIG. 5 is a diagram showing six blocks A to F aligned in a current frame F _n .

6 is a diagram showing a corresponding position of each block in FIG. 5 in a previous frame F _n-1 and an attention area set in the frame.

7 is a diagram showing along with the previous frame F _n-1 region of interest region of interest set in the current frame F _n.

[Explanation of symbols]

Reference Signs List 1 input frame memory unit, 2 subtractor, 3 orthogonal transformation unit, 4 quantization unit, 5 encoding unit, 6 buffer memory unit, 7 inverse quantization unit, 8 inverse orthogonal transformation unit, 9 adder, 1
0 Reference frame memory unit, 11 Motion detection unit, 12
Motion compensation prediction unit, 13 attention area tracking unit, 14 code amount control unit, 15 motion vector, 16 second encoding accuracy instruction, 20 input frame memory output, 21 motion compensation prediction unit output, 22 subtracter output, 23 Output of orthogonal transformation unit, 24 Output of quantization unit, 25 Output of encoding unit, 26 First encoding accuracy instruction, 27 Output of inverse quantization unit, 28 Output of inverse orthogonal transformation unit, 29 Output of adder ,
30 reference frame section output, 31 code amount control section output, 40 attention area of previous frame F _n−1 , 41 attention area of current frame F _n , 130 block position determination section, 1
31 Initial setting unit, 132 Attention area setting unit, 133
Position holding unit.

Claims (8)

[Claims] 1. When a frame image is divided into unit areas, and movement between frames is tracked for each unit area,
Independently of the unit area, the motion of the attention area having an arbitrary shape determined in the frame image is estimated using the tracking result of the movement of the unit area having a predetermined positional relationship with the attention area. Attention area tracking method. 2. The tracking of inter-frame motion performed for each unit area is originally performed independently of the tracking of the attention area as part of motion compensation prediction.
2. The attention area tracking method according to 1. 3. A method for detecting a position of an area corresponding to a unit area provided in a current frame in a previous frame, and using a result of the reverse direction detecting step to obtain a notice provided in the previous frame. A forward direction detecting step of detecting a position of an area corresponding to the area in a current frame, the method comprising: 4. The reverse direction detecting step detects, as a candidate region, a region in which a corresponding region in a previous frame of a unit region aligned with a current frame is likely to be located, and for each unit region, Generating motion information indicating the displacement of the candidate area; the forward direction detecting step selects, from among the candidate areas taken in the previous frame, those included in the attention area in the previous frame as important areas; 4. The attention area tracking method according to claim 3, wherein the attention area is tracked by comprehensively considering the following. 5. The motion information generated in the backward direction detecting step is a motion vector used for motion compensation prediction. The forward direction detecting step averages a motion vector related to the important area to calculate a motion vector of the attention area. 5. The attention area tracking method according to claim 4, wherein the determination is performed. 6. When a frame image is divided into unit areas, and the movement between frames is tracked for each unit area, the movement of the attention area defined in the frame image independently of the unit area is obtained. Estimated by using the tracking result of the motion of the unit area having a predetermined positional relationship with the attention area, and the estimated attention area in the frame image is coded with processing different from that of the other areas. Image encoding method. 7. A reverse direction detecting step of detecting a position of an area corresponding to a unit area provided in a current frame in a previous frame, and using a result of the reverse direction detecting step to obtain a notice provided in the previous frame. A forward direction detecting step of detecting the position of the area corresponding to the area in the current frame, and an encoding step of encoding the area of interest in the frame image with processing different from that of the other areas. Image encoding method. 8. A means for determining a position in a previous frame of a region corresponding to a unit region provided in a current frame, and utilizing the determination result to determine a position of a region corresponding to a region of interest provided in the previous frame. Means for setting the position to the current frame.