JP3498875B2 - Video processing system - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention stably detects a characteristic scene effective for retrieving a moving image from a moving image in which human behavior is recorded, stores the scene, and stores the scene. The present invention relates to a moving image processing system capable of automatically summarizing a moving image based on its location.

[0002]

2. Description of the Related Art When recalling a past experience while watching a moving image, a T (Time)
It is effective to determine P (Place) and W (Who), use them as keywords, and search based on an episodic viewpoint. Conventionally, in order to realize this, a method of first recognizing a person or an object as a search keyword from a moving image and summarizing and managing the moving image based on the recognition result is used. However, even if such a method is used, since it is a moving image that is constantly moving and it is a past record, it is often the case that they do not remember their place, date and time, or name. When there is only one place in a moving image, it is often impossible to identify where it is.

[0003]

As described above, in the past, it was difficult to stably recognize a person or an object from a moving image. By the way, when recalling past experiences together with people and objects, information on a place, that is, "where" is extremely important. However, it is rare for a single image to identify a place in a moving image in which an action is recorded. Therefore, it is difficult to automatically summarize the moving image from the "where" point of view. An object of the present invention is to solve such a conventional problem, stably detect a characteristic scene effective for remembering past experiences, and automatically summarize a moving image from the viewpoint of location. An object of the present invention is to provide a moving image processing system capable of performing the above.

[0004]

In the moving image processing system of the present invention, a moving image input from a camera is targeted.
First, the distance between adjacent frames of a certain image is obtained (width of the image), and then the moving image is divided into a still section and a variable section based on the obtained distance (see FIG. 3). For each frame of the moving image, a region having a characteristic brightness is extracted from the image, and the characteristic region is tracked in an adjacent frame. That is, in the case of a moving image, the information of the luminance remains stronger in the perception than the shape, so that the region having the characteristic luminance is extracted. The still section having the above characteristic areas in common is summarized as one event section. Furthermore, consecutive frames within the fluctuation section having the characteristic region in common are summarized as one event section. In addition, the direction of camera movement is determined in the fluctuation section. In other words, in the changing section, the video cannot be analyzed, and therefore only the direction of movement and the captured object can be tracked. On the other hand, in the still section, a still image that characterizes the section is obtained, and the images are classified based on the mutual similarity of the still images in each event section. Based on the above results, a summary of a plurality of still images and a place represented by a positional relationship between the still images is obtained. In the means for extracting a region having a characteristic brightness from the image for each frame of a moving image, the image is divided into a number of constant rectangular regions, and a histogram of the brightness is obtained in each constant region (Fig. 5). For example, one moving image is divided into 20 × 20 = 400 pixels, and the brightness is set to 0.
(Black) to 255 (white) level. Then, a constant area in which the luminance distribution is concentrated is detected from the histogram (see the dark portion in FIG. 5). Targeting only a certain area in which the distribution of luminance is concentrated, adjacent certain areas having similar luminance are collected and extracted as a characteristic area. In the means for tracking the detected characteristic region in the adjacent frame, the overlapping characteristic regions in the adjacent frames are obtained, and the matching between the overlapping characteristic regions is performed to obtain the movement amount for each region. Further, the movement amount of the entire screen of one frame is obtained from the movement amount of each area, and the movement amount of each area is corrected from the total movement amount.

[0005]

BEST MODE FOR CARRYING OUT THE INVENTION According to the present invention, an image is extracted only from a still section, only a direction of movement is extracted from a moving section, and this is registered, so that information to be remembered later can be recorded without a shortage. . The place is represented by an image of a still section associated with physical position information. The consecutive frames within the variable section having the characteristic region in common are set as one event section, and the section moved by walking is extracted as one event section. Further, it is possible to process the moving section by extracting the area having the same brightness based on the characteristics of the brightness and tracking the extracted area. As a result, it is possible to stably detect a characteristic scene that is effective for retrieving a moving image from a moving image that records past activities, and automatically summarize the moving image from the viewpoint of location. It becomes possible. Note that the place is extremely effective when recollecting past experiences.

[0006]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The principle and embodiments of the present invention will be described in detail below with reference to the drawings. When focusing on the behavior of a person in a moving image, the line of sight often repeats stationary and moving. Of these, pay attention to the place in front of you in the still section. On the other hand, in the moving (fluctuation) section, careful analysis of the image is not performed,
In many cases, they only track the direction of movement or the object in the image. Therefore, the image is extracted only from the still section, only the direction of movement is extracted from the operation section, and this is registered, so that information to be remembered later can be recorded without a shortage. The place can be represented by an image (scene) of a still section associated with physical position information. For example, there is a window image to the right of the desk image. In a scene that moves on foot, for example, a corridor or a ceiling is extracted as a characteristic region. And, when you are going straight, the characteristic region always exists. By setting consecutive frames in the variable section having a characteristic region in common as one event section, a section moved by walking can be extracted as one event section. In addition, it is said that the visual system, which is responsible for the perception of human movement, does not sense color. That is, it is known that information on only brightness is used. It is also said that motion can be perceived when the pictures of adjacent images are slightly moving. Furthermore, tracking of objects and pictures in motion is performed based on information of luminance rather than shape. Therefore, it is possible to process the moving (fluctuation) section by extracting the area having the same brightness based on the characteristics of the brightness and tracking the extracted area.

FIG. 2 is a block diagram of an embodiment in which the present invention is applied to an information management apparatus for video images of daily activities in an office or the like. For example, as shown in FIG. 2, a moving image recorded from the camera 1 and the microphone 2 attached to the operator's head is stored in the memory 30 from the control unit (CONT) 40 via the bus 50. The moving image is, for example, 3
It is sampled and input at 0 frame / second and 640 horizontal × 480 vertical pixels / frame. A program for image processing is stored in the memory 30, and the CPU 10 reads the program from the memory 30 and executes it. The moving image stored in the memory 30 is summarized by the execution of the program, and the result is stored in the memory 30 or the magnetic disk device 20. Also, this summary result is
0 is read from the memory 30 or the magnetic disk device 20 and displayed on the display (CRT) 3 via the interface unit (IF) 60. The user uses the keyboard (KEY) 4 and the mouse 5 to search the displayed summary.

FIG. 1 is a functional block diagram showing an embodiment of a moving image processing program stored in the memory shown in FIG. Normally, the visual system, which controls the perception of human movement, does not perceive color. That is, it is known that human beings use only information on luminance. Further, when tracking an object or a picture in motion, the tracking is performed based on the information on the brightness rather than the shape. Therefore, also in the present embodiment, the moving image is summarized in the time direction based on the luminance feature as follows. In FIG. 1, each of the functional blocks 31 to 37 represents a program, and this program is activated in the order of arrows. The image input from the camera 1 is sent to the inter-image difference processing unit 31 and the luminance-based block extraction processing unit 33. Difference processing unit 3 between images
In 1, every time an image of one frame is sent, the difference processing of the brightness of the image of the adjacent frame is performed. If the difference in luminance is larger than that of the previous frame by the difference processing, it can be seen that it is a moving section. As a result, the moving image can be separated into a still section and a moving section. In the difference processing, for example, the difference value D is calculated by the following equation (1).

[Equation 1] Here, fij (t) is the luminance at the ij coordinate of the t-th frame image, and fij (t-1) is the luminance at the ij coordinate of the t-1th frame image.

FIG. 3 is a diagram showing an example of a difference processing result according to the present invention, and FIG. 4 is a detailed block diagram of a luminance-based block extraction processing section in FIG. Further, FIG. 5 is an explanatory diagram of a method of detecting a characteristic rectangular area based on a luminance histogram. As shown in FIG. 3, the stationary / moving section detection unit 32 sets a section whose difference value is less than or equal to the threshold value θ as a stationary section and a section whose difference value is greater than θ as a moving section. When the moving section is detected, the luminance-based lump extraction processing unit 33 is activated. The luminance-based lump extraction processing unit 33 determines whether the moving section and the still section that is in contact with the moving section have images of 0 to several frames as shown in FIG.
Perform processing as shown in. That is, here, a moving image is divided into, for example, a rectangular area of 20 pixels in the vertical direction and 20 pixels in the horizontal direction (this is referred to as a cell), a histogram of brightness is calculated for each of the cells, and the same histograms are put together into a block. Represent. This is to detect the movement of chunks between frames. In FIG. 4, the luminance histogram calculation unit 3
30 divides the image into cells of horizontal 20 × vertical 20 pixels as described above, and obtains a luminance histogram as shown in FIG. 5 for each cell. The characteristic brightness cell extraction unit 331 determines that the total of the frequencies of four consecutive classes including the maximum frequency class is 3/4 (300) or more of the number of pixels of the entire cell To extract. In FIG. 5, four continuous classes including the maximum frequency class are represented by diagonal lines. The labeling unit 332 collects adjacent cells having the same brightness and extracts clusters having the same brightness, targeting only cells having characteristic brightness. In the lowermost part of FIG. 4, as the luminance-based lumps, five lumps in dark colors are shown in descending order of frequency. Each of these clusters is formed by enclosing cells with characteristic brightness showing the same brightness in a circle or a rectangle.

FIG. 6 is a detailed functional block diagram of the lump tracking processing section in FIG. The lump tracking processing unit 34 detects the movement of each lump, determines the movement in the entire frame, and tracks the movement amount. For this purpose, the characteristic regions that overlap in adjacent frames are obtained, the regions are matched, and the amount of movement for each region is obtained.
The movement of the entire screen of one frame is obtained from the movement amount of each area, and the movement amount of each area is corrected from the total movement amount. By the way, in the perception of human motion, it is said that motion can be perceived when a picture of an adjacent image is slightly moving. The moving image is generally sampled and displayed at 30 frames / sec. This is a sample interval sufficient for the perception of natural motion. Therefore, an object in an image input at 30 frames / sec basically does not move in adjacent frames. That is, they are considered to overlap. In the lump tracking processing unit 34 shown in FIG.
The same chunk candidate detection unit 340 detects chunks that overlap each other in adjacent frames and sets them as candidates for the same chunk. Next, the matching unit 341 performs matching between chunks of the same chunk candidate to obtain the movement amount of the chunk. Here, between the frames sampled at 30 frames / second, the movement amounts of all the objects in the image are almost the same. However, since the lumps are extracted ambiguously, the local matching of the lumps requires different movement amounts between the lumps. For this reason, the correction processing unit 342 adopts the largest number of opinions (takes the majority of different opinions) from the movement amount of each block, and sets it as the movement amount in the entire frame. Then, matching is performed again on the block showing a movement amount that is significantly different from the movement amount based on the movement amount between frames. That is, the matching is performed again only for those having a significantly different movement amount. In that case, if the values do not change much, adjust to the whole. As a method of adopting the largest number of opinions from the movement amount of each block, an existing classification method, for example, a pseudo-collection type classification method is used. This selects the closest movement amount of each chunk and sets both as one class. The amount of movement of the class is, for example, the average value of both. This operation is repeated until the closest movement amount between classes becomes a certain value or more. Then, of the remaining classes, the movement amount of the class having the largest total area of the belonging lumps is adopted.

FIG. 7 is an explanatory diagram of a method of identifying a block of adjacent frames according to the present invention, and FIG. 8 is an explanatory diagram of a method of identifying a block between similarly spaced frames. Here, an example is shown in which lumps A and B are detected at time t, and lumps C and D indicated by diagonal lines are detected at time t + 1. The lump tracking processing unit 34 determines that the lumps A and B are partly combined and moved to the lump C, and the lump B is separated into the lumps C and D and moved. However, in the correction processing unit 342, when the lumps A and lumps C and the lumps B and lumps D have a high degree of similarity in luminance and the lumps B and C have low degrees of similarity in luminance, the lumps B to C are changed. Delete the transition. That is,
It is determined that lump A has moved to lump C and lump B has moved to lump D. here,
The brightness similarity is obtained, for example, by obtaining the correlation value of the brightness histogram shown in FIG. In addition, as shown in FIG. 8, when the lumps separated once are combined again (lump B), the lump tracking processing unit 34 determines that the lumps A and B are similar to each other if the luminosity similarity between the lumps A and B is high. Judge that they are the same lump and delete all lumps A and lumps A to lump B on the way. Such separation / integration of lumps by luminance occurs in an object that artificially generates light, such as a display, but this process can reduce the amount of intermediate results during proper extraction and recording of lumps. it can.

In FIG. 1, a lump tracking processing unit 3
On the basis of the result of No. 4, the stationary section classification unit 35 checks the commonality of the existence of the lumps in the stationary sections before and after the moving section, and when many lumps are shared, both stationary sections are regarded as the same stationary section. Summarize as an event section. The commonality of the lumps can be determined by detecting the amount of the lumps disappeared from the field of view through the moving section, or whether or not the same lump is included in the central region of both stationary sections. In this way, it is possible to summarize the video in several still image columns. Next, the moving section classifying unit 36, similar to the still section classifying unit 35, from the result of the chunk tracking processing unit 34,
First, we examine the sharability of chunks in each frame. Consecutive frames in which the same block exists in a certain area of the image is 1
It is summarized as one event section. The event section summarized by this processing includes a section walking along the corridor. Here, the corridor and ceiling are extracted as the same block in all frames. In other movement sections, the movement direction is obtained. In the case of abrupt movement, the direction of movement is obtained when matching blocks of adjacent frames. If the movement is slow, it can be obtained from the records of the appearance and disappearance of the lumps. For example, during the movement section, a new lump appears from the left,
When the lump disappears from the right, it can be determined that the camera is moving to the left. Further, when the same block exists in the images of the start and end frames of the moving section, it is possible to detect the moving direction of the camera by matching the blocks in both frames.

FIG. 9 is a diagram showing an example of the summary result.
The still section summarizing unit 37 recognizes the still image (for example, the video of the first frame of the still section) obtained by the still section classifying unit 35, and detects the same still image. Through the above processing, still images correlated with each other in the positional relationship shown in FIG. 9 are obtained in the episode storage 38. Figure 9
Represents the scene around the desk. That is, the desk and the window of the person who is placed behind me are first photographed, the edge of my CRT and the keyboard are photographed to the right, the front of the keyboard is photographed to the right, and the right Has a copy of the book on his desk, and the book is on the top of it. By presenting this to the user through the CRT of FIG. 2, it is possible to accurately inform the work place of the work from a certain time point to a certain time point. The result of Figure 9 is
It also serves as a template for recognizing subsequent locations. That is, by matching the input image with the still image of FIG. 9 and its positional relationship as a template, the same place as in FIG. 9 can be recognized. In other words, it can be recognized that it is a desk work.

[0014]

As described above, according to the present invention,
It is possible to stably detect a characteristic scene that is effective for retrieving a moving image from a moving image that records past activities, and to automatically summarize the moving image from the viewpoint of location.

[Brief description of drawings]

FIG. 1 is a functional block diagram of moving image processing of an information management device showing an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of an information management device to which the present invention is applied.

FIG. 3 is a diagram for explaining a method of detecting a still / moving section from a moving image in the inter-image difference processing unit of FIG. 1.

FIG. 4 is a functional block diagram of a lump extraction process based on luminance in FIG.

5 is a diagram for explaining a method of detecting a characteristic rectangular area based on the luminance histogram in FIG.

FIG. 6 is a functional block diagram of a chunk tracking unit in FIG. 1.

FIG. 7 is a diagram for explaining a method of identifying chunks in adjacent frames according to the present invention.

FIG. 8 is a diagram for explaining a method of identifying a block between distant frames according to the present invention.

9 is a diagram showing an example of a summary result stored in the episode memory of FIG. 1. FIG.

[Explanation of symbols]

1 ... Camera, 2 ... Microphone, 3 ... Display, 4 ... Keyboard, 5 ... Mouse, 10 ... CPU, 20 ... Magnetic disk, 30 ... Memory, 40 ... Control unit, 50 ... Bus, 6
0 ... interface unit, 31 ... difference processing unit between images,
32 ... Stationary / moving section detection unit, 33 ... Lump-based block extraction processing section, 34 ... Block tracking section, 35 ... Stationary section classification section, 36 ... Moving section classification section, 37 ... Stationary section summary section , 38 ... Episode storage, 330 ... Luminance histogram calculation unit, 331 ... Characteristic luminance cell extraction unit,
332 ... Labeling unit, 340 ... Same block candidate detection unit, 3
41 ... Matching unit, 342 ... Correction processing unit.

─────────────────────────────────────────────────── ─── Continuation of the front page (56) Reference JP-A-6-333048 (JP, A) JP-A-5-189562 (JP, A) JP-A-7-193748 (JP, A) JP-A-7- 236115 (JP, A) JP 5-37848 (JP, A) JP 61-200789 (JP, A) JP 6-153155 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T ^7/ 00-7/60 H04N 5/91 G06F 17/30

Claims (4)

(57) [Claims] 1. A means for obtaining a distance between images of adjacent frames from a moving image input from a camera, a means for separating the moving image into a still section and a changing section based on the obtained distance, A unit for extracting a region having a characteristic brightness from the image as one lump for each frame, and a lump overlapping the above lump in adjacent frames,
A means for obtaining and tracking the movement amount of the chunk, a means for summarizing a stationary section having the chunk in common as one event section, and a continuous frame in a fluctuation section having the chunk in common as one event section And a means for summarizing the above. 2. Similarity between the means for determining the direction of camera movement in the varying section, the means for determining a still image that characterizes the still section summarized as the one event section, and the still images in each event section. 2. The method according to claim 1, further comprising: a means for obtaining an image and classifying the images, and a means for obtaining a summary of a plurality of still images and a location represented by a positional relationship between the still images based on the result. The moving image processing system described. 3. A means for extracting a region having a characteristic brightness, a means for obtaining a histogram of brightness for each constant area of an image, a means for detecting a constant area in which the brightness distribution is concentrated, Targeting only a certain area where the distribution of
The moving image processing system according to claim 1, further comprising: a unit that collects adjacent constant areas having similar brightness. 4. A means for tracking the characteristic area in adjacent frames, a means for obtaining a characteristic area that overlaps in an adjacent frame, and a matching between the overlapping characteristic areas to determine a movement amount for each area. 2. The method according to claim 1, further comprising: a means for obtaining the movement amount of the entire screen of one frame from the movement amount of each area, and a means for correcting the movement amount of each area from the movement amount of the entire area. Image processing system.