JPH05108795A - Picture processor - Google Patents

Abstract

PURPOSE:To efficiently perform a processing extracting an object from a picture and make it possible to easily perform a picture processing even by a user who has no knowledge of the picture processing. CONSTITUTION:In a picture processor extracting an arbitrary object from an inputted picture, a TV camera 5 imaging the face of a user 6, a display part 4 displaying an inputted picture, a line of sight detection part 3 detecting a line of sight from an outputted picture from the TV camera 5, a line of sight analysis part 2 classifying the movement of the line of sight into a hopping eye movement and a following eye movement by analyzing the movement of the line of sight detected by the line of sight detection part 3 and discriminating whether an object is moving or is still and a picture processing part 1 switching the extraction method of the object according to this discrimination result.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for detecting a movement of a user's line of sight and extracting an object (region) in an image based on the movement.

In image analysis and image understanding, the process of separating an object in an image from a background (a part other than the object) is
This is one of the most frequently used important processes. However, there are various methods for realizing this, and the optimum method and parameters differ depending on the image to be handled and the object. For this reason, when trying to extract an object from an arbitrary image, not only knowledge of the image processing method and characteristics of the object is required, but also trial and error selection must be performed. There are many.

If the parameters need to be set once, such as the quality inspection of a single product, the expert providing the system may first give the parameters. However, when considering a system that allows users to interactively process images regardless of the type or purpose of the image to be processed, even non-specialized users should be aware of the method and parameter selection. Without this, an interface that can easily perform those settings is needed.

For example, when an image such as a commercial film is to be produced, various types of images such as still images, moving images, characters, animations, and CGs are handled, and they are freely divided / composed and edited. , I have to make a video. At this time, the purpose of the user of this system is to create an image, and it is meaningless to know the parameters necessary for extracting a part of the image.

In other words, setting parameters and methods is not the purpose for the user, and if this is complicated, the user will be forced to do extra work and is inappropriate as an interface. is there.

The present invention realizes a man-machine interface that can be handled even by a person who does not have specialized knowledge or that assists method selection in an image processing apparatus for interactively extracting an object from an image. ..

[0007]

2. Description of the Related Art Conventionally, various image processings have been performed in an image processing apparatus. Among them, for example, in order to extract an object from an input image, the image is first divided (segmentation).

This is to divide an image into regions having uniform features by using features such as color, shade, texture (a repeating pattern formed by elements being arranged according to a certain rule).

This processing includes edge detection (detection of discontinuous portions of features) and area division (division of image into connected areas of features).
Done by Many methods have been proposed for both processes, but which feature and which method to use are
It depends on the image to be handled.

However, in any of the methods, it is impossible to divide the image without error, and it is necessary to combine the methods and use the model and knowledge of the object.

[0011]

SUMMARY OF THE INVENTION The above-mentioned conventional device has the following problems. (1) In conventional image processing, in order to extract an object from an image, the characteristics of the image and the object must be known.

(2) In the process of extracting an object from an image, when giving a characteristic of the object, what kind of characteristic should be given and what method should be used in order to separate the object by image processing. Knowledge of whether to choose is required.

Therefore, a person who does not have specialized knowledge cannot use the image processing apparatus as described above. The present invention solves such a conventional problem, efficiently performs the process of extracting an object from an image, and enables a user who has no knowledge of image processing to easily perform the image processing. The purpose is to

[0014]

FIG. 1 is a principle view of the present invention, in which 1 is an image processing unit, 2 is a line-of-sight analysis unit, 3 is a line-of-sight detection unit, 4 is a display unit, 5 is a TV camera. (TV camera), 6 shows a user.

In order to solve the above problems, the present invention has the following configuration. (1) In an image processing device that extracts an arbitrary object from an input image, a line-of-sight detection unit 3 that detects the line of sight of the user 6, and a line-of-sight analysis unit that analyzes the movement of the line of sight detected by the line-of-sight detection unit 3. 2 and the image processing unit 1 that extracts an arbitrary object from the input image based on the analysis result of the line-of-sight analysis unit 2.

(2) In the line-of-sight analysis unit 2 of the above configuration (1), the movement of the line-of-sight is analyzed to classify the eye movements into compliant eye movements and jumping eye movements. (3) In the line-of-sight analysis unit 2 of the configuration (2), it is discriminated whether the object is stationary or exercising by classifying it into compliant eye movements and jumping eye movements, and image processing is performed. The method of extracting the object in the part 1 is switched.

(4) In the line-of-sight analysis unit 2 of the configuration (3), except for the jumping eye movement, the average movement speed (V _ave ) of the line of sight is obtained only for the compliant eye movement, and from the average movement speed, I tried to determine whether the object is stationary or in motion.

(5) The line-of-sight analysis unit 2 of the above configuration (3) detects the speed and the position of the line of sight when the line of sight is moving due to the consequent eye movement, and sends the detection information to the image processing unit 1. Therefore, the image processing unit 1 uses the above detection information to extract the moving object on the image from the moving image.

[0019]

The operation of the present invention based on the above construction will be described with reference to FIG. The input image to be processed is displayed on the display unit 4, and the user 6 works while watching this display screen.
At this time, the TV camera 5 images the face of the user 6 and sends the image to the line-of-sight detection unit 3.

The line-of-sight detection unit 3 detects the line-of-sight movement of the user 6 from the input image. Then, the visual line analysis unit 2 analyzes the movement of the visual line detected by the visual line detection unit 3. In this analysis, the speed of movement of the line of sight during a certain period is obtained based on the characteristics of the eye movement, and the speed is classified into compliant eye movement and jumping eye movement.

In this case, the jumping eye movement occurs not only when looking at a stationary object but also when looking at a moving object, but the compliant eye movement has a characteristic that occurs only when focusing on the moving object. Therefore, it is determined whether the object is moving or stationary depending on whether or not the compliant eye movement is included, excluding the jumping eye movement.

Based on this determination result, the processing method of the image processing unit 1 is selected, and the object is extracted according to the information such as the position and speed of the line of sight. The determination as to whether the object is moving or stationary is performed by the line-of-sight analysis unit 2 by determining an average movement speed only for the compliant eye movements and determining whether the average movement speed is larger or smaller than a predetermined value. To do.

Then, when it is determined that the object is moving, the method of the moving object extracting process of the image processing unit 1 is selected,
The moving object is extracted using the detection information (information such as speed and position) from the line-of-sight analysis unit 2. When it is determined that the target object is still, the stationary object extraction processing method is selected and the stationary target object is extracted.

As described above, by detecting the line of sight and analyzing the movement of the line of sight, it is possible to obtain information such as the position and movement of the object, thereby switching the image extraction processing method. Therefore, the user can extract the necessary target object only by looking at the image regardless of the image to be handled and the target object.

[0025]

Embodiments of the present invention will be described below with reference to the drawings. 2 to 7 are views showing an embodiment of the present invention, FIG. 2 is an example of eye movement (A is a case of gazing at a stationary object, B is a case of gazing at a moving object), and FIG. 4 is a block diagram of the line-of-sight detecting unit, FIG. 5 is a block diagram of the line-of-sight position accumulating unit, FIG. 6 is a block diagram of the line-of-sight velocity calculating unit and the motion discriminating unit, and FIG. 7 is an average velocity calculating process. It is a flowchart.

In the figure, the same symbols as in FIG. 1 indicate the same components.
Further, 7 is a delay circuit, 8 is a line-of-sight position accumulation unit, 9 is a line-of-sight velocity calculation unit, 10 is a motion determination unit, 11 is a switch circuit, 1
2 is a moving object extraction processing unit, 13 is a stationary object processing unit, 14 is a face image input unit, 15 and 16 are feature point extraction units, 17 is a face posture calculation unit, 18 is a line-of-sight vector calculation unit, and 19 is a gazing point. Calculation unit, 20 is a speed calculation unit, 21 is a speed accumulation unit, 22
Is an average speed calculation unit, 23 is a speed determination unit, 24 is an exercise flag, and 25 is an exercise determination unit.

First, an example of eye movement will be described with reference to FIG. In FIG. 2, A represents the movement of the line of sight when viewing a stationary object. Usually, when a person (user) looks at a stationary object, he or she gazes at one point for about 250 msec (see FIG. 2A), and then makes a jumping motion (see FIG. 2A) for several tens of msec to move the line of sight. Repeating that, the information of the object is obtained. This jumping movement is called jumping eye movement.

The displacement by one jumping motion is 3 ° to 6 °
(Visual angle), which is several 100 ° / sec in terms of speed. On the other hand, the compliant eye movement is an eye movement that occurs when gazing at a moving object, and the line of sight moves smoothly with the movement of the object (see FIG. 2B).

Jumping eye movements occur not only when looking at a stationary object but also when looking at a moving object (FIG. 2B).
However, it is said that the consequent eye movement occurs only when attention is paid to a moving object, and the velocity is about 5 ° / sec.

Based on the characteristics of these eye movements, the speed of movement of the line of sight during a certain period of time is obtained, and it is discriminated whether the eye movements are compliant or jumping. Depending on how, it is determined whether the object is moving or stationary.

Then, the image processing method is selected, and the object is extracted according to the information such as the position and speed of the line of sight. Next, the image processing apparatus of this embodiment will be described. As shown in FIG. 2, the image processing apparatus includes an image processing unit 1, a visual line analysis unit 2, a visual line detection unit 3, a display unit 4, a TV camera 5, and a delay circuit 7.

The image processing unit 1 includes a switch circuit 11, a moving object extraction processing unit 12, and a stationary object extraction processing unit 13.
The visual axis analysis unit 2 is provided with a visual line position storage unit 8, a visual line velocity calculation unit 9, and a motion determination unit 10.

The TV camera 5 is used by the user 6 of the image processing apparatus.
Is a camera that is arranged in front of the camera and captures an image of the face of the user 6. The line-of-sight detection unit 3 inputs the image captured by the TV camera and detects the movement of the eyes of the user 6.

The line-of-sight analysis unit 2 inputs the information detected by the line-of-sight detection unit 3 and analyzes the movement of the line-of-sight. The image processing unit 1 performs a process of extracting an object from the input image. The delay circuit 7 is a circuit that delays an image input to the image processing unit 1 for a predetermined time.

The switch circuit (SW) 11 performs a switching operation based on the output signal of the motion discriminating unit 10 so that the input signal to the image processing unit 1 is transferred to the moving object extraction processing unit 12 or the stationary object extraction processing. It is a circuit for sending to the unit 13.

During operation of the image processing apparatus shown in FIG.
The input image to be processed is displayed on the image display unit 4, the face of the user 6 who observes the image is captured by the TV camera 5, and the image is input to the line-of-sight detection unit 3. The line-of-sight detection unit 3 sequentially detects the line of sight of the user 6 from the input image.

In this case, the line-of-sight detection unit 3 is, for example, as shown in FIG. 4, the face image input unit 14, the feature point extraction units 15, 1
6, a face posture calculation unit 17, a line-of-sight vector calculation unit 18, a gazing point calculation unit 19, and the like. The output image of the TV camera 5 is input to the face image input unit 14 and then the characteristic points are extracted by the characteristic point extracting units 15 and 16.

Further, the face posture calculating section 17 calculates the face posture using the feature point information extracted by the feature point extracting section 15. The line-of-sight vector calculation unit 18 calculates the line-of-sight vector using the output information of the face posture calculation unit 17 and the output information of the feature point extraction unit 16.

Finally, the gazing point calculation unit 19 calculates the gazing point of the user 6 from the calculation result of the line-of-sight vector calculation unit 18. In this way, when the line of sight of the user 6 is detected,
The detection result is output to the line-of-sight analysis unit 2. In the line-of-sight analysis unit 2, the input data (data of the position of the line of sight of the user 6)
Is stored in the line-of-sight position storage unit 8.

After that, the line-of-sight velocity calculation unit 9 calculates the line-of-sight velocity, and based on the velocity, it is determined whether the object of interest is moving or stationary. The line-of-sight position storage unit 8 has a capacity of storing N number of line-of-sight positions sampled at a time period (several hundred msec) substantially equal to the cycle in which the jumping eye movement occurs.

For example, the storing method is as shown in FIG.
The data is shifted every time the position of the line of sight is input,
The new data is written at the end. The oldest data (D _tN ) output from the line-of-sight position storage unit 8 by shifting is output to the image processing unit 1.

In the example of FIG. 5, assuming that one line-of-sight data is D _{t + 1} , D _t , D _t-1 , D _t-2 ... D _{t- N} ,
The line-of-sight position storage unit 8 has D _t , D _t-1 , D _t-2, ... D _t.
N pieces of data consisting of _tN are stored. In this case, _assuming that the period of the jumping eye movement is T _s and the sampling period of the line of sight is T, there is a relationship of N = T _s / T.

Now, when the data D _{t + 1} is input in this state, all the data are shifted by one (shifted to the right in the figure), and the oldest data D _tN is output.
In this way, it shifts each time data is input,
The oldest data is sequentially output.

The line-of-sight velocity calculator 9 and the motion discriminator 10 are
For example, it is configured as shown in FIG. That is, the line-of-sight velocity calculation unit 9 includes a velocity calculation unit 20, a velocity accumulation unit 21, and an average velocity calculation unit 22, and the motion determination unit 10 includes a velocity determination unit 23, a motion flag 24, and a motion determination unit 25. ..

First, the speed calculation unit 20 of the line-of-sight speed calculation unit 9
In the line-of-sight position data storage unit 8, the latest data D _t and the previous data D _t are stored.
The moving speed V _t of the line of sight during _t-1 is calculated, and this is input to the speed accumulation unit 21.

In this case, the moving speed V _{t of the} line of sight is obtained by the following equation.

[0047]

[Equation 1]

In the above equation, X _t-1 and X _t are components in the X-axis direction, and Y _t-1 and Y _t are components in the Y-axis direction. Further, the velocity storage unit 21 has the same configuration as the line-of-sight position storage unit shown in FIG. 5, and when data is input, all data is shifted and the oldest data is output.

The speed determination unit 23 of the motion determination unit 10 fetches the moving speed V _t input to the speed storage unit 21,
It is determined whether the velocity V _t is the velocity of the jumping eye movement. In this determination, if V _t > V _th0 , jumping eye movement is performed, and if V _t ≦ V _th0 , resting or compliant eye movement is determined. Here, V _th0 is the limit velocity of the saccadic eye movement, and is a value of V _th0 = 5 ° / sec to 10 ° / sec.

When it is not the jumping movement, the velocity V _t is input to the average velocity calculating section 22 and the bit of the movement flag 24 is set. The movement flag 24 is a shift register, and shifts by 1 bit each time one data is input.
The average speed calculator 22 sequentially adds the speeds V _t input from the speed determiner 23, and subtracts the speed V _tN output from the speed accumulator 21 when the bit of the flowed motion flag is set.

Then, the sum of the speeds is divided by the number of bits of the current motion flag to obtain the average motion speed V _ave of the line of sight excluding the jumping motion. This speed V _ave is input to the motion determination unit 25 of the motion determination unit 10 and the image processing unit 1.

In the motion determining section 25, the input velocity V
Determine the motion of the object from _ave . Since the velocity V _ave does not include jumping eye movements, the velocity V _ave should be almost zero when the object is stationary.

Therefore, if the velocity V _ave > V _th1 (V _th0 is a reference value and a + value close to 0), it is determined that the object is moving and 1 is input to the image processing unit. Otherwise,
Assuming that the user is stationary, 2 is input (the reason why the average speed is used for the motion determination is that there is slight movement of the eyeball even when looking at a stationary object).

The process for obtaining the average motion velocity V _ave will be described with reference to the process flowchart of FIG.
The process numbers in FIG. 7 are shown in parentheses. First, the data of the moving speed V _t of the line of sight calculated by the speed calculating unit 20 is
When input to the speed accumulation unit 21, the data in the speed accumulation unit 21 is shifted (shifted to the right in the figure), and the oldest data is output to the average speed calculation unit 22. Further, the moving speed V _t
Is input to the speed determination unit 23. At this time, the motion flag 24 is shifted to the left by 1 bit (S1).

After that, the speed determination unit 23 determines whether the input speed is
Degree V_tBut V_t> V_th0Or V_t≤V _th0Determine
(S2). This judgment result is V_t≤V_th0If so, stand still
Or, since the eye movement is compliant, the maximum movement flag 24
The lower bit is incremented by +1 (S3), and the speed V_tData of
It is output to the average speed calculator 22 and added to add up the speed sum V_sum
Is calculated (S4).

If V _{t ≤V th0 is} not _satisfied (S2),
The above processing (S3, S4) is not performed. After that, if the carry f _c of the motion flag 24 is f _c = 1 (S5), the average speed calculation unit 22 subtracts the speed V _tN output from the speed storage unit 21 (adds −V _tN ) to _obtain the sum of the speeds. V _sum is calculated (S6).

In the above process (S5), f _c =
If it is not 1, the process of S6 is not performed. Finally, divide the sum V _sum at a rate of the number of bits n of "1" in the motion flag _{24 (V su m / n)} , determines the average motion velocity V _ave of sight excluding the jumping movement (S7).

As shown in FIG. 3, the image processing section 1 includes a moving object extraction processing section 12, a stationary object extraction processing section 13, and a switch circuit 1.
1 and switches the switch circuit 11 according to the value output from the motion discriminating unit 10 of the line-of-sight analyzing unit 2.

The input image is the same as the image given to the display unit 4, but is input to the image processing unit 1 via the delay circuit 7 which delays by the time required to calculate the line-of-sight velocity. Then, in the moving object extraction processing unit 12 or the stationary object extraction processing unit 13 switched by the switch circuit 11,
The extraction process of the object is executed based on the information on the moving speed and the position of the line of sight.

[0060]

As described above, the present invention has the following effects. That is, since the information of the target object is obtained from the line of sight, the user can extract the target object only by looking at the image. Therefore, even a user who has no knowledge of image processing can easily perform image processing, and the performance of the man-machine interface in the image processing apparatus can be improved.

[Brief description of drawings]

FIG. 1 is a principle diagram of the present invention.

FIG. 2 is a diagram showing an example of eye movements (A is a case of gazing at a stationary object and B is a case of gazing at a moving object) according to an embodiment of the present invention.

FIG. 3 is a configuration diagram of an image processing apparatus.

FIG. 4 is a configuration diagram of a line-of-sight detection unit.

FIG. 5 is a configuration diagram of a line-of-sight position accumulation unit.

FIG. 6 is a configuration diagram of a gaze velocity calculation unit and a motion determination unit.

FIG. 7 is an average movement speed calculation processing flowchart.

[Explanation of reference numerals] 1 image processing unit 2 visual line analysis unit 3 visual line detection unit 4 display unit 5 TV camera 6 user

Claims (5)

[Claims] 1. In an image processing apparatus for extracting an arbitrary object from an input image, a line-of-sight detection unit (3) for detecting the line-of-sight of a user (6) and a line-of-sight detected by the line-of-sight detection unit (3). A line-of-sight analysis unit (2) for analyzing the movement of the human body and an image processing unit (1) for extracting an arbitrary object from the input image based on the analysis result of the line-of-sight analysis unit (2) are provided. Image processing device. 2. The image processing apparatus according to claim 1, wherein the line-of-sight analysis unit (2) analyzes the movement of the line of sight to classify the eye movement into compliant eye movements and jumping eye movements. 3. The line-of-sight analysis unit (2) classifies the subject's eye movements as jumping eye movements and jumping eye movements to determine whether the object is stationary or in motion, and an image processing unit. The image processing apparatus according to claim 2, wherein the method of extracting the object in (1) is switched. 4. The line-of-sight analysis unit (2) excludes jumping eye movements, and only for compliant eye movements,
Obtains an average speed of motion of the line of sight (V _ave), the average than motion velocity (V _ave), or the object is stationary, motion and it was characterized by either determining and 3. The image processing apparatus according .. 5. The line-of-sight analysis unit (2) detects the velocity and the position of the line of sight when the line of sight is moving due to compliant eye movement, and sends the detection information to the image processing unit (1). The image processing apparatus according to claim 3, wherein the image processing unit (1) extracts the moving object on the image from the moving image using the detection information.