JP2000187478A - Picture processor and picture processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a picture processor and a picture processing method capable of realizing picture expression according to movements of an object contained in animation data, and easily creating and outputting a still picture permitting to intuitively grasp the movements of the object. SOLUTION: This picture processor comprises an animation data taking part for extracting plural frame pictures from animation data and storing them in a frame memory, a central picture setting part for selecting a specific frame picture from the plural frame pictures stored in the frame memory and setting it as a central picture, a movement detecting part for detecting the movement of the object contained in the plural frame pictures stored in the frame memory, a trajectory picture creating part for processing the plural frame pictures and creating a trajectory picture based on the movement of the central picture and the object, and a picture synthesis part for creating a still picture where the moving state of the object is expressed by synthesizing and processing the central picture and the trajectory picture.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method, and more particularly, to processing of synthesizing moving picture information taken by a video camera, an electronic still camera or the like into one still picture (still picture). The present invention relates to an image processing apparatus and an image processing method for displaying and outputting images.

[0002]

2. Description of the Related Art In recent years, the spread and function of AV equipment have been remarkable. In particular, sophisticated image recording devices such as video cameras and electronic still cameras have come to be used in various scenes including consumer and business use. Images taken by these devices can be easily displayed and output on a monitor of a television or a personal computer (hereinafter abbreviated as a personal computer) via a liquid crystal panel, a cable, or the like mounted on the device body, Further, it is also possible to print out directly via a printer or the like. However, of the various types of captured image information, the amount of moving image information is enormous compared to that of still images, and a series of shooting contents such as the movement of the subject can be grasped only after reproducing and displaying the entire moving image. Storage, playback,
In the output, there is a problem that a large amount of storage capacity is required, and it takes a long time to check the photographed contents, and a problem that an arbitrary image such as a still image cannot be easily output. I have.

[0003] On the other hand, with the spread of image recording equipment and the improvement of image processing technology, needs for various methods of expressing captured image information are increasing. For example, a series of movements of a subject can be intuitively grasped by expressing a trajectory of a relatively fast-moving subject, such as a vehicle running or a golf swing, as a still image. 2. Description of the Related Art Conventionally, moving image information is used as a method of expressing general moving image information (including video information of a television broadcast) as a still image without being limited to a moving image captured by a video camera, an electronic still camera, or the like. A method of extracting and displaying one frame image displayed at a specific time as a representative image from a plurality of frame images, or extracting a plurality of frame images as a representative image at regular time intervals, There is known a method of displaying representative images sequentially or collectively on a divided screen.

FIG. 17 is a conceptual diagram of a technique for extracting and displaying one image from a plurality of frame images constituting moving image information. In FIG. 17, it is assumed that the moving image data VD includes an image in which the vehicle travels from left to right in the drawing. Here, since the moving image data VD can be considered to be composed of a plurality of frame images (still images) arranged in time series, for example, a specific time By extracting and outputting one frame image F1 at T4, the motion of the subject (vehicle C1) can be included in the still image. However, in such an expression method,
The frame image F1 extracted as a representative is a specific time T4
Since there is only one still image, the motion immediately before and after the specific time T4 can be objectively estimated, but a series of motions of the subject cannot be grasped. That is, even if one frame image F1 including the traveling vehicle is displayed as a still image, the movement of the vehicle C1 is not accurately represented, and the vehicle C1 may not move depending on conditions such as the background image and the moving speed of the vehicle. Recognized as you are.

FIG. 18 is a conceptual diagram of a method of extracting a plurality of frames from a plurality of frame images constituting moving image information and displaying the extracted plurality of frames at a time on a divided screen. As shown in FIG. 18, for example, four frame images F1, F2, F3, and F4 at times T2, T4, T6, and T8 are extracted at regular time intervals from a series of frame images representing the motion of a subject. , The four frame images F1, F2, F3, and F4 can be viewed in chronological order even though they are still images, and a series of movements of the subject (the traveling of the vehicles C1 to C4) can be performed. Status) can be grasped. However, in such an expression method, the viewer looks over the plurality of frame images extracted as the representative image over the entire area, and the viewer estimates the relationship between the frame images to determine the vehicle C1 to C4.
In addition, depending on any one of the four frame images F1 to F4, a series of movements of the subject can be intuitively grasped in the same manner as described above. I can't. That is, even if a plurality of frame images F1 to F4 including the traveling vehicle are displayed as still images sequentially or collectively, there is a large part of the movement of the vehicle that is left to the guess of the viewer, and the vehicle is accurately recognized. Not necessarily. Such a problem always occurs when moving image information of a subject having a relatively fast movement, such as a golf swing, is represented as a still image, not limited to a traveling vehicle.

[0006]

By the way, as a method of expressing moving image information as a still image, a plurality of frame images are extracted from video data, and they are simply multiplexed and combined to form one still image. The method is disclosed in JP-A-10-2
90450. This method selects and extracts a plurality of frame images from video data at equal time intervals, calculates an average value of pixel values at the same position in each frame, and uses the average value as a pixel value of a multiplex still image. However, since there is a problem that the pixel value of the moving part becomes small and becomes almost invisible, the pixel value is given a predetermined weight and further improved so that the moving part does not become blurred. It is.

However, the method described in the above publication corrects all the extracted frame images so that the pixel values are uniformly expressed without blurring of the pixel values, and is merely a simple superimposition method. As shown in FIG.
Different times T2, T for traveling vehicles C1 to C4
4, the frame images F1 to F4 in T6 and T8 are simply obtained by multiplexing them into one still image FQ. In other words, although it is possible to recognize the motion trajectory and trajectory of the subject, it does not accurately represent the characteristics of the motion, for example, a sense of speed, and further, an image that can intuitively grasp the motion of the subject, An image having a special effect cannot be obtained.

Therefore, the present invention realizes an image expression corresponding to the movement of a subject included in moving image data, and easily generates a still image from which the movement of the subject can be grasped intuitively.
It is an object of the present invention to provide an image processing device and an image processing method capable of outputting.

[0009]

An image processing apparatus according to claim 1, wherein a plurality of frame images are extracted from moving picture information and stored in a storage means, and said image information is stored in said storage means. Reference image setting means for selecting a specific frame image from a plurality of frame images and setting the reference image as a reference image; and motion detecting means for detecting a motion of a subject included in each of the plurality of frame images stored in the storage means A trajectory image generating unit configured to process the plurality of frame images based on the reference image and the detected motion of the subject to generate a trajectory image expressing a motion trajectory of the subject; and the reference image. Image synthesizing means for synthesizing the trajectory image and the trajectory image to generate a trajectory composite image. The image processing apparatus according to claim 2, wherein the motion detecting means detects at least a motion speed of the subject, and the trajectory image generating means generates different trajectory images. Wherein the method of generating the locus image is changed based on the operation speed.

[0010] The image processing apparatus according to the third aspect is the first aspect.
Or the image processing apparatus according to 2, further comprising: a reference image selection unit that selects an arbitrary frame image from the plurality of frame images stored in the storage unit, wherein the reference image setting unit selects the reference image by the image selection unit. The obtained frame image is set as the reference image. The image processing device according to claim 4 is the image processing device according to claim 1 or 3, wherein the trajectory image generation unit includes:
A plurality of trajectory image generation methods for generating different trajectory images are provided, and further, trajectory image selection means for selecting an arbitrary generation method from the plurality of trajectory image generation methods is provided. An image processing device according to a fifth aspect of the present invention is the image processing device according to the first, second, third, or fourth aspect, further including an image output unit that displays and outputs the trajectory composite image generated by the image composition unit. It is characterized by the following.

According to a sixth aspect of the present invention, there is provided an image processing apparatus, comprising: storage means for storing moving image information comprising a plurality of frame images; and selecting a specific frame image from the plurality of frame images stored in the storage means. Generating a trajectory image expressing a motion trajectory of a subject included in each of the plurality of frame images using a reference image setting unit to be set as a reference image and the plurality of frame images stored in the storage unit A trajectory image generating means and an image synthesizing means for synthesizing the reference image and the trajectory image to generate a trajectory composite image are provided. 8. The image processing apparatus according to claim 7, wherein the storage unit stores moving image information including a plurality of frame images, and a specific frame image is selected from the plurality of frame images stored in the storage unit and set as a reference image. Reference image setting means, a motion detection means for detecting a motion of a subject included in each of the plurality of frame images stored in the storage means, and a motion of the subject detected by the motion detection means. ,
Image processing means for processing the reference image.

An image processing method according to claim 8 is
Extracting a plurality of frame images from the moving image information and storing the extracted frame images in a storage unit; selecting a specific frame image from the plurality of frame images stored in the storage unit and setting the selected frame image as a reference image; Detecting the movement of the subject included in each of the plurality of frame images stored in the means, the reference image, and, based on the detected movement of the subject, processing the plurality of frame images, Generating a trajectory image representing the motion trajectory of the subject; and combining the reference image and the trajectory image to generate a trajectory composite image.

[0013]

Embodiments of the present invention will be described below with reference to the drawings. <First Embodiment> FIG. 1 is a block diagram showing a first embodiment of an image processing apparatus according to the present invention. In FIG. 1, reference numeral 10 denotes a moving image data capturing unit (image information capturing means);
0 is a frame memory (storage means), 30 is an MPU, 40
Is a hard disk, 50 is a monitor (image output means) such as an LCD, 60 is a printer (image output means), 70 is an input section such as a key switch (reference image selection means, locus image selection means), and 80 is data / command transmission. Bus. Here, the MPU 30 includes a motion detection unit (motion detection means) 31
, A trajectory image generating unit (trajectory image generating unit) 32, a center image setting unit (reference image setting unit) 33, and an image synthesizing unit (image synthesizing unit) 34.

Each of these functions is generally as follows. (1) Moving Image Data Capture Unit The moving image data capture unit 10 extracts a plurality of frame images from the moving image data and stores them in a frame memory 20 described later. Here, the moving image data may be composed of video data or a plurality of continuously shot still images. In short, any frame image may be used as long as it is composed of a succession of a plurality of frame images, and the captured frame image may be an analog image or a digital image. The schematic configuration of the moving image data capturing unit 10 will be described with reference to FIG. In FIG. 2, 1
1 is a frame image selection unit, 12 is an analog-digital converter (hereinafter, referred to as A / D), 13 is an image compression unit, 1
4 is a bus interface.

The frame image selecting section 11 is provided with an MP
In accordance with a command from U30, frame images are selected at predetermined time intervals from the moving image data, and an image signal of each selected frame image is extracted. Here, the selected frame image may be all the frame images constituting the moving image data, or may be frame images at a specific time interval, for example, every 1/5 sec or 1/10 sec. It is necessary to have a time interval enough to grasp the movement of the subject included in the moving image data. The time interval used for selecting a frame image may be a method using a reference value preset by the MPU 30 or an arbitrary value instructed via the input unit 70 by a user of the image processing apparatus. Various methods such as a method can be set. When the selected frame image is an analog signal, the A / D 12 converts the analog image signal into a digital image signal.

The image compression section 13 converts the image signal of each frame image converted into a digital image signal by the A / D 12 or extracted as a digital image signal by the frame image selection section 11 into a predetermined image compression standard. Process as follows. The JPEG standard or the like can be applied as a frame image compression encoding method. here,
JPEG (Joint Photographic Coding Experts Group)
Is a standard for compressing and encoding original image data by a method such as DCT (Discrete Cosine Transform), quantization, variable length coding, etc., and is an international standard adopted for color facsimile apparatuses, electronic still cameras, etc. It is. In addition, as a compression encoding method of image information, generally, in addition to JPEG,
Since various formats and standards such as GIF, TIFF, LHA, and ZIP are used, an appropriate method can be adopted according to the embodiment. Bus interface 14
Converts the compressed frame image into the transmission width of the bus 80 and transfers it to the frame memory 20.

(2) Frame Memory 20 The frame memory 20 is a DRAM (Dynamic Random Accumulator).
ess Memory) and the like, and the moving image data capturing unit 10
Is associated with header information such as image identification information and the like, and is stored in the image data storage area designated by the MPU 30. FIG. 3 is a conceptual diagram showing an internal area of the frame memory 20. As shown in FIG. 3, the internal area of the frame memory 20 is roughly divided into a format table area, an information table area, an image data area, and an offset area. The format table area stores format information that is comprehensive information on image information. The information table area includes a header for identifying image information such as image information identification information including number information for identifying image information, and time information indicating a position (time) of moving image data in a time series. Information is stored. The image data area stores compression-encoded frame images (compressed image data), and the offset area stores offset data (blank) for setting the data length of the compressed image data in the image data area to a fixed length. . As described above, each piece of compressed image data is stored in the image data area in association with the header information stored in the information table area.

(3) Motion Detection Unit 31 The motion detection unit 31 detects a feature amount indicating the motion of the subject from a plurality of frame images stored in the frame memory 20. Here, the feature amount indicating the motion of the subject is information indicating a motion locus or trajectory of the subject, an operation (movement) speed, an operation (movement) direction, and the like, and can be generally represented as vector information. As a method of motion detection,
A block matching method that divides frame images at different times into block regions, performs block matching processing on the reference image for each of the same block regions, and detects a motion between subjects of a frame from a coordinate position where an error is minimized. Can be applied. The block matching method is based on ISO / IECJTC1 / SC29 / W
It is widely used in IS11172-2 standard and the like, which was internationally standardized in G11. The block matching method will be described later.

(4) Central Image Setting Unit 33 The central image setting unit 33 selects and extracts a specific frame image from a plurality of frame images stored in the frame memory 20, and performs central image setting in image synthesis processing described later. It is set as. Here, the center image is an image serving as a reference when synthesizing images, and may be an image located at the beginning or end of a series of frame images, or may be a frame image at a specific time. . in short,
Any device may be used as long as it extracts a nucleus (original) image to be combined with a trajectory image created by the trajectory image generation unit 32 described later.

(5) Trajectory Image Generation Unit 32 The trajectory image generation unit 32 includes a feature amount indicating the motion of the subject detected by the motion detection unit 31, and a center image setting unit 3
A plurality of frame images stored in the frame memory 20 are processed based on the subject image included in the center image extracted by 3 to generate a trajectory image representing a motion trajectory or an afterimage corresponding to the motion of the subject. Things. The trajectory image is generated by determining the subject from a plurality of frame images stored in the frame memory 20 based on the detection result of the motion detection unit 31 and extracting the contour component of the subject from each frame image other than the center image. A unique trajectory image that expresses the motion of the subject by performing a process of multiplexing the extracted contour components or a process of generating an afterimage area based on the contour components, and further performing a gradation process and adjusting the density of the image. Create

(6) Image Synthesizing Unit 34 The image synthesizing unit 34 synthesizes the center image extracted by the center image setting unit 33 with the trajectory image created by the trajectory image generating unit 33 to generate a trajectory synthesized image. Output. (7) MPU 30 and Hard Disk 40 The hard disk 40 stores programs executed by the MPU 30 and data necessary for operation. Therefore, MP
The U30 executes the application program stored in the hard disk 40 to execute the motion detection unit 31, the trajectory image generation unit 32, and the center image setting unit 3 described above.
3. The functions of the image synthesizing unit 34 are realized by software, and a series of image processing and memory management, which will be described later, and output control to the monitor 50 and the printer 60 are performed.

(8) Monitor 50, Printer 60 This device displays or prints out a locus composite image synthesized by the image synthesizing unit 34, and is an output device such as a monitor of a television or a personal computer, or a printer. Here, in FIG. 1, the monitor 50 and the printer 60 are shown as being configured to be directly connected to the bus 80, but the present invention is not limited to this, and a communication interface or the like connected to the bus 80 may be used. It may be a facsimile device, a personal digital assistant (PDA), a personal computer, or the like, which is connected via a communication line via the communication line. In the description of the present invention, in addition to the display output to the monitor 50 and the print output to the printer 60, the entire operation of outputting the trajectory composite image as a still image is referred to as “display output” for convenience. (9) Input Unit 70 The input unit 70 is various key switches provided in the image processing apparatus. The input unit 70 executes an application program by the MPU 30, performs image processing, and monitors the monitor 50 and the printer 60.
A control signal, such as a display output to the CPU, is generated. Further, it also has a function as a reference image selecting unit or a locus image selecting unit in the embodiment described later. When the present invention is implemented by a personal computer or the like, various input devices such as a keyboard, a mouse, and a pen tablet are included in addition to dedicated key switches provided in the image processing device.

Next, the processing operation in the image processing apparatus having the above configuration will be described with reference to the drawings.
FIG. 4 is a flowchart illustrating a processing operation of the image processing apparatus according to the present embodiment. First, an outline of the processing operation will be described with reference to the above-described configuration, and then each step will be individually described. As shown in FIG.
In 101 and S102, the moving image data capturing unit 10 selects and extracts a plurality of frame images from input moving image data at predetermined time intervals, and performs a compression encoding process so as to conform to a standard such as JPEG. Frame memory 20
In a predetermined storage area. At this time, header information for identifying the compressed frame image is stored in association with the header information. Next, in step 103, the central image setting unit 33 sets a central image as a reference in image synthesis from among the plurality of frame images stored in the frame memory 20, and in steps S104 and S105, the motion detecting unit 31 As a result, an object (moving object) accompanying an operation included in a plurality of frame images stored in the frame memory is detected, and information indicating the operation characteristics of the object is output.

Next, in steps 106 and 107, the trajectory image generation unit 32 extracts the contour components of the subject included in each frame image based on the above-described motion characteristics of the subject and the center image, and extracts them. The contour component is multiplexed or a residual image area is generated based on the contour component, and further, a processing process for adjusting the density of the contour component and a gradation process are performed to create a trajectory image expressing the motion of the subject. Next, Step 108, S109
, The image combining unit 34 combines the center image and the trajectory image into one still image, and outputs the obtained still image to the monitor 50, the printer 60, or the like. As described above, the processing operation of the image processing apparatus according to the present embodiment includes:
It roughly comprises a moving image data capturing step, a center image setting step, a motion detecting step, a trajectory image creating step, and an image synthesizing step.

Hereinafter, each step will be described with reference to the drawings. (1) Moving Image Data Import Step FIG. 5 is a conceptual diagram showing the moving image data import step.
Hereinafter, a vehicle running as a subject will be described as an example.
Moving image data captured by a video camera or an electronic still camera, or moving image data such as video information of a television broadcast is a set of a series of frame images arranged in time series. As shown in
At predetermined time intervals from a series of frame images, for example, times T2 and T2 corresponding to time-series positions of the moving image data VD.
6, T10,... Frame images Fa, Fb, Fc,.
Are selected and extracted by the moving image data capturing unit 10. The selected and extracted frame image is the moving image data VD
May be all of a series of frame images constituting
The time interval is not limited as long as the movement of the subject, that is, the running state of the vehicle can be grasped from the extracted frame image. Extracted frame image Fa,
Fb, Fc,... Are converted into digital image signals in order to easily execute image processing to be described later, and JPEG is used in order to effectively use the storage capacity of the frame memory 20.
And the like, and are sequentially stored in the image data area in association with image identification information, time information, and the like.

(2) Center Image Setting Step In this step, a plurality of frame images F stored in the frame memory 20 by the center image setting section 33 are set.
From among a, Fb, Fc,..., for example, the frame image Fb at time T6 is selected, and header information indicating that the frame image is the center image is added to the information table area to set the frame image as the center image. As described above, the center image is a reference image at the time of image synthesis, and the extraction method is to extract an image located at a start end, an end, or an intermediate time among the captured frame images. May be set in advance, or based on a detection result of a motion characteristic of a subject, which will be described later, for example, a vehicle whose traveling vehicle is located at the center position of the frame image may be selected.

(3) Motion Detection Step In this step, the motion detection unit 31 detects a traveling vehicle from a plurality of frame images stored in the frame memory 20 and calculates the trajectory or trajectory of the motion. Hereinafter, a method for detecting the movement of the subject included in the image information will be described with reference to the drawings.
First, block matching will be described as an example of a technique applied to motion detection. FIG. 6 is a conceptual diagram illustrating a block matching method applied to the motion detection step. For example, see “Television Society,“ Image Information Compression ”, Ohmsha, p. 92, 1991] describes a block matching method for detecting a motion of a subject included in moving image data composed of a plurality of continuous frame images.

According to the above-mentioned document, as shown in FIG. 6, a frame image of interest (hereinafter referred to as a current frame image for convenience) Fn and a frame image at the immediately preceding time (hereinafter referred to as a previous frame image for convenience). Pattern matching is performed on blocks (regions) Bn and Bm at specific positions in Fm. In the pattern matching method, for example, an absolute value sum S of a difference between a pixel Xn in a block Bn and a pixel Xn-i in a block Bm obtained by shifting the pixel Xn by i pixels is calculated according to the following equation. That is, the shift amount i that minimizes the evaluation amount is searched to detect the motion amount. S = {| Xn-Xn-i |... Here, the sum calculation in the equation is executed for all the pixels Xn belonging to the block Bn. in this way,
In the block matching method, the current frame image is divided into blocks, and a similar position to the previous frame image is searched for by block matching in block units to detect the movement of the subject included in the frame image, such as the movement speed and direction. .

As a method applied to motion detection, a method of calculating a motion vector from a change in a characteristic component of a subject region included in a frame image can be applied in addition to the above-described block matching method. . In this method, first, a Laplacian process is performed on a luminance component included in a frame image, a zero crossing point of the processed image is detected as a region boundary line, and a portion closed by a continuous boundary line is defined as an independent region (subject region). ). Alternatively, a color component included in the frame image is analyzed, and a continuous portion having a small change in hue is replaced with a representative color and extracted as one region. Then, a feature component of the extracted region, for example, the position of the center of gravity of the region is detected, and the amount of change between frame images is calculated, thereby obtaining a motion vector indicating the amount of parallel movement of the entire region. Therefore, by outputting the calculated change amount as the motion parameter of the region, the value (movement amount, direction) of the motion vector can be obtained by calculation at an arbitrary pixel position in the subject region. Further, as still another method of motion detection, a large number of contour components in a frame image are detected,
It is also possible to apply a method of collecting a large number of detected contour lines into groups and detecting the motion of the entire image by neural network processing.

(4) Trajectory Image Creation Step In this step, the trajectory image generation unit 32 expresses the trajectory representing the running state of the vehicle based on the plurality of frame images stored in the frame memory 20 and the central image. Generate an image. FIG. 7 is a conceptual diagram illustrating a first example of the trajectory image creation step. As shown in FIG. 7, a plurality of frame images Fa and F stored in the frame memory 20 are displayed.
A locus image is generated by extracting only contour components of the vehicle from b and Fc based on the motions (movement characteristics) of the vehicles Ca, Cb and Cc determined in the above-described motion detection step and arranging them in a time-series manner. Is done. The trajectory image generated here is different from the mere multiple synthesis of the frame images described in the related art, in that only the outline component is synthesized, so that the trajectory image accurately represents the entire movement of the vehicle. Further, in the center image setting step, processing of a locus image is performed based on the position of the vehicle Cb included in the frame image (for example, Fb) set as the center image.

More specifically, the brightness, saturation, and the like of a contour component (hereinafter, referred to as a reference contour component) constituting an image (subject image) of the vehicle Cb included in the center image Fb are defined as reference values.
Of the plurality of contour components constituting the trajectory image, the brightness and saturation of those located near the reference contour component are approximated to the reference value, and as the distance increases, the brightness and saturation are reduced, and so-called blur is reduced. Adjust to the state that occurred. At this time, the processing is performed on only the contour components constituting the trajectory image located on the opposite side (rear side; left side in the drawing) of the traveling direction of the vehicle Cb, and is located on the traveling direction side (front side; right side in the drawing). The contour component is deleted. That is, the locus image La is created by performing image processing using the contour components of the frame image belonging to the past than the center image. The above-described trajectory image creation processing is referred to as “first trajectory image creation processing”.

As another method, in addition to the above-described multiple composition processing of the contour components, a specific image element expressing the motion direction and the motion state of the subject, for example, a line component extending in the motion direction of the subject is generated and arranged. 7, a locus image Lb may be created as shown in FIG.
The above-described trajectory image creation processing is referred to as “second trajectory image creation processing”. In the above-described processing of the trajectory image, a method of extracting, multiplex-synthesizing, the contour components of the subject included in the frame image, and further performing the processing of each contour component based on the position of the subject included in the center image is described. As described above, the center image may be processed so that the entire area formed by the contour components constituting the trajectory image is subjected to gradation processing. In this case, M
A functional unit (image processing means) for processing the central image is provided in the PU 30, and the density becomes lighter as the distance from the subject is increased, based on the density of the area on the reference contour component side. A more specific embodiment is not limited to the above-described example as long as it allows a viewer of a combined or processed image to accurately recognize the operation state of a subject.

Next, a second example of the trajectory image creation step will be described. In the above-described first example, the description has been given on the assumption that the moving image data is obtained by capturing a state in which the vehicle included in the extracted frame image travels with respect to the fixed background. There is a case where the moving vehicle is moving image data in which the moving vehicle is fixed at a fixed position in the frame image. A process of creating a locus image in such a case will be described. FIG. 8 is a conceptual diagram illustrating a second example of the trajectory image creation step. As shown in FIG. 8, from a plurality of frame images Fi, Fj, and Fk extracted from the moving image data and stored in the frame memory 20, the background components Gi, Only Gj and Gk are extracted. This background component G
i, Gj, and Gk are easily generated by deleting, from the frame images Fi, Fj, and Fk, the subject (vehicles Ci, Cj, and Ck) components whose motion has not been detected in the motion detection step. Then, the extracted background component G
i, Gj, and Gk are arranged in time series to generate a trajectory image. Further, in a center image setting step, a background component (hereinafter referred to as a reference background component) included in a frame image (for example, Fk) set as the center image The processing of the locus image is performed based on Gk.

More specifically, the luminance, the saturation, and the like of the reference background component Gk are processed so as to represent the largest degree of blur, and the reference background component Gk of the plurality of background components constituting the trajectory image is processed. The locus image Lc is created by approximating the object located in the vicinity of the reference value with the reference value and performing a blurring process so that the luminance and the saturation of the background component are adjusted as the distance increases. Note that, similarly to the first example described above, the trajectory image creation processing includes not only the synthesis and processing of the background component, but also the addition processing of the line component expressing the moving (flowing) state of the background component and the gradation processing. May be used alone or in combination.

(5) Image Synthesizing Step In this step, the image synthesizing unit 34 synthesizes the center image and the trajectory image to generate a still image expressing the running state of the vehicle. Display output via 9 and 10 are conceptual diagrams showing a first example of the image synthesizing step. Here, FIG.
Locus image La created by the locus image creation process of
FIG. 10 is a conceptual diagram illustrating a case where the center image is combined with the center image. FIG. 10 is a conceptual diagram illustrating a case where the center image is combined with the track image Lb created by the above-described second track image creation processing. FIG.

As shown in FIGS. 9 and 10, the frame image F set as the center image in the center image setting step
b and the trajectory images La and Lb created by the trajectory image creation step are synthesized to obtain the still images ED1 and ED2 in which the trajectory images La and Lb expressing the traveling state of the vehicle Cb are synthesized. That is, the center image F
b is only a specific one of a series of frame images constituting the moving image data. However, in the track images La and Lb, the contour component and the region are processed based on the position of the vehicle Cb included in the center image Fb. Is performed, the traveling trajectory of the vehicle Cb is represented in one frame image Fb by combining the two. Then, the synthesized still images ED1 and ED2 are transferred to the monitor 50 or the printer 60 via the bus 80 and displayed and output.

FIG. 11 is a conceptual diagram showing a second example of the image synthesizing step, and shows a case where the locus image Lc created by the locus image creating process shown in FIG. 8 and the image of the vehicle Ck are synthesized. ing. As shown in FIG. 11, the image component of the vehicle Ck included in the frame image Fk set in the center image in the center image setting step and the trajectory image Lc created in the trajectory image creation step are multiplexed and synthesized. A still image ED3 representing the running state of the vehicle Ck in the background flow is obtained. That is, the first
Similarly, the center image Fk is only a specific one of a series of frame images constituting the moving image data, but the trajectory image Lc has an outline or a contour based on the background component Gk included in the center image Fk. Since the area has been processed,
By combining the two, the traveling state of the vehicle Ck is represented by a background flow in one frame image Fk.
According to the present embodiment having the above series of steps, by detecting a series of movements of the subject in the moving image data, creating a trajectory image corresponding to the position and movement of the subject, and performing synthesis processing with the center image, A still image in which the movement of the subject is accurately represented can be created. Therefore, since the motion of the moving object can be intuitively grasped with one still image, it can be effectively used as an index of moving image data.

<Second Embodiment> Next, a second embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings. In the present embodiment, in the first embodiment described above, the motion (movement characteristics) of the subject detected by the motion detection unit 31 is analyzed, and the method of creating the locus image is changed according to, for example, the traveling speed of the vehicle. It is like that. That is, in the image processing apparatus illustrated in FIG. 1, the trajectory image generation unit 32 analyzes the detection result such as the operation speed and the operation direction of the subject detected by the motion detection unit 31,
It has a control function for changing the creation processing of the trajectory image.
Specifically, as shown in FIG. 9, in the still image ED1 in which the locus image La created by the first locus image creation process is synthesized with the center image Fb, the movement of the vehicle Cb is recognized to be relatively slow. While FIG.
As shown in the above, in the still image ED2 obtained by synthesizing the locus image Lb created by the second locus image creating process with the center image Fb, the motion of the vehicle Cb is expressed so as to be recognized relatively quickly. . Therefore, based on such characteristics, the operation speed of the subject detected by the motion detection unit 31 is determined, for example, by comparing it with a predetermined threshold value to determine whether the operation is a low-speed operation or a high-speed operation. The first and second trajectory image creation processes are appropriately switched.

Here, the relationship between the moving speed of the subject and the trajectory will be described. When the subject is moving at a low speed, the outline of the moving object is relatively clearly grasped. The low-speed operation state can be expressed by performing multiple synthesis for the number of frame images without blurring the outline of the subject (moving object) included in each frame image in accordance with the characteristics, so that the motion of the subject can be appropriately recognized. be able to. In the present invention, in addition to such multiple synthesis of the contour components, the brightness and saturation of the contour near the subject included in the center image are approximated to those of the center image so that the line segments and colors are clearly displayed. In a contour far from the subject, processing is performed to blur the line segments and colors, and a processing process for expressing the movement of the subject in a more realistic manner is performed. On the other hand, when the subject moves at a high speed, the outline of the moving object is extremely vaguely grasped.
By blurring the outline of the subject included in each frame image mutually according to the human visual characteristics, a high-speed operation state can be expressed, and the movement of the subject can be appropriately recognized.

Hereinafter, the processing operation of this embodiment will be described with reference to the flowchart of FIG. Steps equivalent to those in the above-described first embodiment will be simplified. As shown in FIG. 12, step S201,
In S202, a plurality of frame images are selected and extracted from the moving image data at predetermined time intervals, compression-encoded, and stored in a predetermined storage area of the frame memory 20.
Next, in step 203, the frame memory 20
In step S204, a subject (moving object) included in the plurality of frame images is detected by applying the above-described block matching method or the like, and the Calculate motion characteristics such as motion speed and direction. And step S
In steps S205 to S207, the calculated operation speed is compared with a preset threshold speed. If the calculated operation speed is lower than the threshold value, it is determined that the subject is a low-speed moving object. Then, the above-described first trajectory image creation processing is selected, and processing is performed so that the contour components constituting the trajectory image are expressed relatively clearly. On the other hand, if the calculated operation speed is higher than the threshold value, it is determined that the subject is a high-speed moving object, and the above-described second trajectory image creation processing is selected, and in addition to the multiple synthesis of the contour components, , A processing for superimposing a line component or the like expressing a motion direction or the like is performed.
Next, in Steps 208 and 209, the central image and the above-mentioned trajectory image are combined to form one still image, which is displayed and output via the monitor 50 and the printer 60.

According to the present embodiment having the above-described series of steps, the method of creating the locus image is changed based on the detected series of movements of the subject, particularly, the operation speed (movement speed), so as to correspond to the operation speed. Since the trajectory image creation processing for creating the trajectory image can be executed, a still image that can intuitively recognize the movement of the subject can be created. In the present embodiment, the case where the first and second trajectory image creation processes are used as the processing method selected according to the operation speed of the subject has been described. The method of processing the background component shown in FIGS. 8 and 11 may be changed. In short, any method other than the above may be used as long as it allows a viewer of the synthesized still image to accurately recognize the operation state of the subject. Also, as the processing for determining the operation speed of the subject, a case has been described where comparison processing is performed with one preset threshold value. However, multi-step determination using a plurality of threshold values may be performed, The brightness and the saturation may be weighted using the speed as it is.

<Third Embodiment> Next, a third embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings. In the present embodiment, selection and setting processing of the center image in the center image setting step is changed according to a command input from the input unit 70 or the like. That is, in the image processing apparatus shown in FIG. 1, the central image setting unit 33 sends the central image setting unit 33 to the frame memory 20 in accordance with the central image selection command input from the input unit 70 or via a communication line or the like (not shown). It has a function of extracting an arbitrary frame image from a plurality of stored frame images and setting it as a center image. Specifically, a plurality of frame images stored in the frame memory 20 are displayed and output via the monitor 50 or the printer 60, and an instruction for specifying a frame image to be selected as the center image is input to the user of the image processing apparatus. Prompt. The user browses the displayed and output frame images, selects an arbitrary frame image from the displayed frame images, and selects an input unit 70 such as a key switch or a keyboard.
The center image selection command is input via the
0 is added to the information table area of the image to indicate that the image is the center image.

Therefore, the user inputs, for example, a frame image that seems to be most appropriate for expressing the operation state of the subject or a frame image that is most appropriate for producing a special effect from the input unit 70 or the like. Thus, it can be easily set as the center image. Hereinafter, the processing operation of the present embodiment will be described with reference to the flowchart of FIG. Steps equivalent to those in the above-described first or second embodiment will be simplified.

As shown in FIG. 13, step S301,
In S302, a plurality of frame images are selected and extracted from the moving image data at predetermined time intervals, subjected to compression encoding processing, and stored in a predetermined storage area of the frame memory 20.
Next, in step 303, the frame memory 20
All or a part of the plurality of frame images stored in the printer are displayed and output via the monitor 50, the printer 60, or the like, and the user is prompted to select a frame image to be set as the center image and to input a command. Here, the display output method of the frame images by the monitor 50 and the printer 60 may be a method of sequentially outputting the frame images in time series, or a method of outputting a plurality of frame images collectively on the divided screen. There may be. Further, the frame images to be displayed and output may be all the frame images stored in the frame memory 20, or may be some frame images. That is, the frame images Fa, Fb, F extracted in FIG.
In c, all the frame images may be displayed and output, or only some of the odd-numbered frame images Fa and Fc may be displayed and output. In short, what is necessary is just to display and output a frame image that can be used as a center image in the image synthesis processing.

Next, in step S304, header information is set so that the frame image selected and designated by the user becomes the center image. Further, step S
In 305, a subject (moving object) included in a plurality of frame images by applying the above-described block matching method or the like.
And outputs the operation state of the subject. In step S306, a predetermined trajectory image is generated based on the motion state of the subject and the central image. In steps S307 and 308, the central image and the trajectory image are combined to form one still image, and The display is output via the printer 50 or the printer 60. According to the present embodiment having the above series of steps, instead of uniquely selecting and setting the center image from a plurality of frame images,
Since the user can browse and display and output the frame image and designate and set an arbitrary frame image, it is possible to realize more various image expressions while accurately expressing the movement of the subject. Note that the trajectory image creation step is not limited to the method described in the first embodiment, and may be a method in which the processing method is appropriately changed according to the motion characteristics of the subject as described in the second embodiment. Good.

<Fourth Embodiment> Next, a fourth embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings. In the present embodiment, the change of the trajectory image creation processing in the above-described second embodiment is not controlled based on the motion characteristics of the subject, but is performed by the user of the image processing apparatus via the input unit 70 or the like. It can be set arbitrarily. That is, in the image processing apparatus shown in FIG.
It has a function of arbitrarily selecting a trajectory image creation method from 0 or according to a trajectory image creation processing selection command input via a communication line or the like (not shown).
The process of creating a trajectory image to be selected includes the process of creating a trajectory image according to the motion characteristics of the subject, that is, the first and second trajectory image creation processes, the gradation process, and the background component processing process. In addition, it is possible to arbitrarily select a process of creating a special trajectory image that emphasizes a subject included in the center image or characterizes the subject.

Here, an example of the processing of creating and synthesizing a special trajectory image that emphasizes or characterizes a subject will be described with reference to the drawings. FIG. 14 is a conceptual diagram illustrating an example of a special trajectory image creating process, and FIG. 15 is a conceptual diagram illustrating an example of a combining process of a special trajectory image and a center image. As shown in FIG. 14, only a contour component of the vehicle is extracted from a plurality of frame images Fa to Fe stored in the frame memory 20 based on the motion of the vehicles Ca to Ce determined in the motion detection step, and the time series is extracted. (Multiple composition), a trajectory image expressing the entire movement of the vehicle is generated. Then, based on the position and the contour component of the vehicle Cc included in the frame image (for example, Fc) set as the center image, the contour components located in front of and behind the vehicle Cc blur as the distance from the vehicle Cc increases. The locus image LQ is created by performing processing to generate the locus image.

Next, as shown in FIG. 15, by combining the frame image Fc set as the center image and the special trajectory image LQ, the vehicle Cc in the entire motion trajectory of the vehicles Ca to Ce EDQ is obtained. Accordingly, the user can easily set a desired still image by inputting and setting, for example, a method of creating a trajectory image that is most suitable for producing a special effect via the input unit 70 such as a key switch or a keyboard. Can be created. Hereinafter, the processing operation of the present embodiment will be described with reference to the flowchart of FIG. Steps equivalent to those in the above-described first to third embodiments will be simplified.

As shown in FIG. 16, step S401,
In S402, a plurality of frame images are selected and extracted from the moving image data at predetermined time intervals, subjected to compression encoding processing, and stored in a predetermined storage area of the frame memory 20.
Next, at step 403, the frame memory 20
The center image is set from among the plurality of frame images stored in the image processing unit. In step S404, the subject (moving object) included in the plurality of frame images is detected by applying the above-described block matching method or the like. Outputs the operating status. Then, in steps S405 and S406, a predetermined trajectory image is generated based on the trajectory image creation method selected and set by the user.
In steps 7 and 408, the central image and the trajectory image are combined to form one still image, and the monitor 50 and the printer 60
Display output via. According to the present embodiment having the above series of steps, since the method of creating a locus image can be arbitrarily selected and set according to the user's will, a special locus image created using the motion locus of the subject Is synthesized with the center image, a still image exhibiting a special effect can be easily created. Note that the image processing apparatus of the present invention can be favorably applied to image processing apparatuses such as video players, facsimile apparatuses, and printers, in addition to the above-described video cameras, electronic still cameras, and personal computers. Nor.

[0050]

According to the first, fifth, sixth, seventh or eighth aspect of the present invention, a trajectory image centered on the subject included in the frame image set as the reference image and corresponding to the motion of the subject is generated. Since it is generated, it is possible to easily create and output a still image that can intuitively recognize the movement of a subject captured as moving image information. In particular, the operation state of the subject is changed by multiplexing the contour components of the subject, further clearly displaying the contour components near the subject in the reference image, and performing processing so that the further away from the subject, the more blur or blur occurs. Creates a still image that accurately reflects and displays it on a TV monitor or PC monitor, or
It can be printed out by a printer or the like. Therefore, the moving image information can be expressed as a still image by a simple method and in various expression formats, and can be effectively used for purposes such as index information of the moving image information.

According to the second aspect of the present invention, the moving speed of the subject included in the frame image extracted from the moving image information is detected, and the method of generating the locus image is changed according to the detection result. Accordingly, it is possible to generate a locus image corresponding to the operation speed of the subject. Therefore, since the sense of speed expressed by the locus image differs depending on the operation speed of the subject, just by looking at the created still image,
It is possible to intuitively grasp the movement speed of the subject together with the entire movement. According to the third aspect of the present invention, a frame image serving as a reference image can be arbitrarily selected and set by a user of the image processing apparatus,
Since a locus image centered on the subject included in the reference image and according to the motion of the subject is generated, it is possible to easily create a still image emphasizing the subject located at an arbitrary position. The movement of the subject can be accurately grasped by various expression forms. Claim 4
According to the described invention, a method of generating a trajectory image can be arbitrarily selected and set according to the user's will of the image processing apparatus, so that a wider variety of expression forms can be realized and special effects can be achieved. Can be easily created. In particular, by providing, as a method of generating a locus image, blurring of a contour component, gradation processing, addition of a specific image expressing speed, and the like, a locus image desired by the user can be generated.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a first embodiment of an image processing apparatus according to the present invention.

FIG. 2 is a block diagram illustrating a schematic configuration of a moving image data capturing unit.

FIG. 3 is a conceptual diagram showing an internal area of a frame memory.

FIG. 4 is a flowchart illustrating a processing operation of the image processing apparatus according to the first embodiment.

FIG. 5 is a conceptual diagram showing a moving image data capturing step.

FIG. 6 is a conceptual diagram illustrating a block matching method applied to a motion detection step.

FIG. 7 is a conceptual diagram showing a first example of a trajectory image creation step.

FIG. 8 is a conceptual diagram showing a second example of the trajectory image creation step.

FIG. 9 is a conceptual diagram showing a case where a locus image created by a first locus image creation process and a center image are combined.

FIG. 10 is a conceptual diagram showing a case where a locus image created by a second locus image creating process and a center image are combined.

FIG. 11 is a conceptual diagram showing a second example of the image synthesizing step.

FIG. 12 is a flowchart illustrating a processing operation of the image processing apparatus according to the second embodiment.

FIG. 13 is a flowchart illustrating a processing operation of the image processing apparatus according to the third embodiment.

FIG. 14 is a conceptual diagram showing an example of a special trajectory image creation process applied to the fourth embodiment.

FIG. 15 is a conceptual diagram illustrating an example of a process of combining a special trajectory image and a center image applied to the fourth embodiment.

FIG. 16 is a flowchart illustrating a processing operation of the image processing apparatus according to the fourth embodiment.

FIG. 17 is a diagram showing a concept of a method of extracting and displaying one image from a plurality of frame images constituting moving image information.

FIG. 18 is a diagram showing the concept of a method of extracting a plurality of frames from a plurality of frame images constituting moving image information and displaying them collectively on a divided screen.

FIG. 19 is a conceptual diagram showing a conventional method of expressing moving image information as a still image.

[Explanation of symbols]

Reference Signs List 10 moving image data capturing section (image information capturing means) 20 frame memory (storage means) 30 MPU 31 motion detecting section (motion detecting means) 32 track image generating section (track image generating section) 33 central image setting section (reference image) Setting means) 34 image combining section (image combining means) 40 hard disk 50 monitor (image output means) 60 printer (image output means) 70 input section (reference image selecting means, locus image selecting means) 80 bus

Continued on the front page (51) Int.Cl. ⁷ Identification code FI Theme coat II (reference) G09G 5/00 530 G06F 15/66 450 F term (reference) 5B050 EA06 EA19 FA02 FA03 5B057 CC03 CE08 DA16 DC16 5C080 BB05 DD01 DD21 EE19 EE29 EE32 JJ01 JJ02 JJ05 JJ07 5C082 AA27 BA20 BA41 BB15 BB26 CA55 CB01 CB06 DA26 MM05

Claims (8)

[Claims] An image information capturing unit for extracting a plurality of frame images from moving image information and storing the extracted frame images in a storage unit; selecting a specific frame image from the plurality of frame images stored in the storage unit; Reference image setting means for setting as an image; movement detecting means for detecting movement of a subject included in each of the plurality of frame images stored in the storage means; movement of the subject detected by the movement detecting means A trajectory image generating means for generating a trajectory image representing the motion trajectory of the subject, and an image synthesizing means for generating a trajectory composite image by synthesizing the reference image and the trajectory image. An image processing apparatus characterized by the above-mentioned. 2. The motion detection unit detects at least an operation speed of the subject, and the trajectory image generation unit includes a plurality of trajectory image generation methods for generating different trajectory images, based on the operation speed. The image processing apparatus according to claim 1, wherein a method of generating the trajectory image is changed. 3. The image processing apparatus according to claim 1, further comprising: a reference image selecting unit configured to select an arbitrary frame image from the plurality of frame images stored in the storage unit. The image processing apparatus according to claim 1, wherein an image is set as the reference image. 4. The trajectory image generation means includes a plurality of trajectory image generation methods for generating different trajectory images, and further includes a trajectory image selection means for selecting an arbitrary generation method from the plurality of trajectory image generation methods. The image processing apparatus according to claim 1, wherein the image processing is performed. 5. The image processing apparatus according to claim 1, further comprising an image output unit configured to display and output the trajectory synthesized image generated by the image synthesis unit. 6. A storage means for storing moving image information comprising a plurality of frame images, and a reference image setting means for selecting a specific frame image from the plurality of frame images stored in the storage means and setting the selected frame image as a reference image Using the plurality of frame images stored in the storage unit, a trajectory image generation unit that generates a trajectory image expressing a motion trajectory of a subject included in each of the plurality of frame images; An image processing apparatus comprising: an image synthesizing unit that synthesizes the trajectory image to generate a trajectory composite image. 7. A storage unit for storing moving image information comprising a plurality of frame images, and a reference image setting unit for selecting a specific frame image from the plurality of frame images stored in the storage unit and setting the selected frame image as a reference image A motion detection unit that detects a motion of a subject included in each of the plurality of frame images stored in the storage unit; and the reference image based on the motion of the subject detected by the motion detection unit. An image processing apparatus, comprising: image processing means for processing. 8. A step of extracting a plurality of frame images from moving image information and storing the extracted plurality of frame images in a storage unit, and selecting a specific frame image from the plurality of frame images stored in the storage unit and setting the selected frame image as a reference image. Detecting the movement of a subject included in each of the plurality of frame images stored in the storage unit; and determining the plurality of frames based on the reference image and the detected movement of the subject. Processing the image to generate a trajectory image representing the motion trajectory of the subject; and synthesizing the reference image and the trajectory image to generate a trajectory composite image. Image processing method.