JP3603629B2 - Image processing apparatus and image processing method - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing apparatus and an image processing method, and more particularly, to an image processing apparatus that combines moving image information captured by a video camera, an electronic still camera, or the like into one still image (still image), and outputs the resulting image. And an image processing method.
[0002]
[Prior art]
2. Description of the Related Art In recent years, the spread of AV equipment and the improvement of functions have been remarkable. In particular, sophisticated image recording devices such as video cameras and electronic still cameras have been used in various scenes, including consumer and business use. Images taken by these devices can be easily displayed and output on a monitor of a television or a personal computer (hereinafter abbreviated as a personal computer) via a liquid crystal panel or a cable mounted on the device body, Further, it is also possible to print out directly via a printer or the like.
However, of the various types of captured image information, the amount of moving image information is enormous compared to still images, and a series of shooting contents such as the movement of the subject can be grasped only after reproducing and displaying the entire moving image. It requires a large amount of storage capacity for storing, reproducing, and outputting image information, and takes a long time to check the shooting contents. There is a problem that an image cannot be easily output.
[0003]
On the other hand, with the spread of image recording devices and improvements in image processing techniques, needs for various methods of expressing captured image information are increasing. For example, a series of movements of a subject can be intuitively grasped by expressing a trajectory of a relatively fast-moving subject as a still image, such as a vehicle running or a golf swing.
2. Description of the Related Art Conventionally, not only a moving image captured by a video camera, an electronic still camera, or the like but also general moving image information (including television broadcast image information) is expressed as a still image by forming moving image information. A method of extracting and displaying a single frame image displayed at a specific time from a plurality of frame images as a representative image, or extracting a plurality of frame images as a representative image at regular time intervals, There is known a method of displaying the representative images sequentially or collectively on a divided screen.
[0004]
FIG. 17 is a conceptual diagram of a method of extracting and displaying one image from a plurality of frame images constituting moving image information.
In FIG. 17, it is assumed that the moving image data VD includes an image in which the vehicle travels from left to right in the drawing. Here, since the moving image data VD can be considered to be composed of a plurality of frame images (still images) arranged in time series, for example, a specific time By extracting and displaying and outputting one frame image F1 at T4, the motion of the subject (vehicle C1) can be included in the still image.
However, in such an expression method, since the frame image F1 extracted as a representative is only one still image at the specific time T4, it is not possible to objectively estimate the movement immediately before and after the specific time T4. Although it is possible, it cannot grasp a series of movements of the subject. That is, even if one frame image F1 including the traveling vehicle is displayed as a still image, the movement of the vehicle C1 is not accurately represented, and the vehicle C1 may not move depending on conditions such as the background image and the moving speed of the vehicle. You will be recognized as you are.
[0005]
FIG. 18 is a conceptual diagram of a method of extracting a plurality of frames from a plurality of frame images constituting moving image information and collectively displaying them on a divided screen.
As shown in FIG. 18, for example, four frame images F1, F2, F3, and F4 at times T2, T4, T6, and T8 are extracted at regular time intervals from a series of frame images representing the motion of a subject. , The four frame images F1, F2, F3, and F4 can be viewed in chronological order in spite of being still images, and a series of movements of the subject (running of the vehicles C1 to C4) can be performed. Status) can be grasped.
However, in such an expression method, a plurality of frame images extracted as the representative images must be viewed over the whole, so that the viewer must estimate the association between the frame images and recognize the movement of the vehicles C1 to C4. Further, depending on any one of the four frame images F1 to F4, a series of movements of the subject cannot be intuitively grasped as in the case described above. That is, even if a plurality of frame images F1 to F4 including a running vehicle are displayed as still images sequentially or collectively, there is a large portion of the movement of the vehicle that is left to the guess of the viewer, and the vehicle is accurately recognized. Not necessarily.
Such a problem always occurs when moving image information of a subject having a relatively fast movement such as a golf swing is expressed as a still image, not limited to a traveling vehicle.
[0006]
[Problems to be solved by the invention]
Meanwhile, as a method of expressing moving image information as a still image, a method of extracting a plurality of frame images from video data and simply multiplexing them to create one still image is disclosed in Japanese Patent Application Laid-Open No. H10-290450. It is described in the gazette. This method selects and extracts a plurality of frame images at equal time intervals from video data, calculates an average value of pixel values at the same position in each frame, and uses the average value as a pixel value of a multiplex still image. However, since there is a problem that the pixel value of the moving part becomes small and becomes almost invisible, the pixel value is given a predetermined weight and further improved so that the moving part does not become blurred. It is.
[0007]
However, the method described in the above publication corrects all extracted frame images so that pixel values are uniformly expressed without blurring, and is merely a simple superimposition. For example, FIG. As shown in the figure, frame images F1 to F4 at different times T2, T4, T6, and T8 relating to traveling vehicles C1 to C4 are simply multiplexed into one still image FQ. That is, although it is possible to recognize the motion locus and trajectory of the subject, it does not accurately represent the motion characteristics, for example, a sense of speed, and the like, and an image that can intuitively grasp the motion of the subject, An image having a special effect cannot be obtained.
[0008]
Therefore, the present invention provides an image processing apparatus that realizes an image expression according to the movement of a subject included in moving image data, and that can easily generate and output a still image that allows intuitive grasp of the movement of the subject. And an image processing method.
[0009]
[Means for Solving the Problems]
2. The image processing apparatus according to claim 1, wherein a plurality of frame images are extracted from the moving image information and stored in a storage unit, and a specific frame image is obtained from the plurality of frame images stored in the storage unit. And a reference image setting unit for setting as a reference image, and a subject included in each of the plurality of frame images stored in the storage unit.Feature amount indicating a motionMotion detection means for detectingA trajectory image generation function selection unit that selects a trajectory image generation function to be executed from a plurality of different trajectory image generation functions based on the operation speed of the subject, among the feature amounts detected by the motion operation detection unit; running is selected by the trajectory image generating function selection means and the trajectory image generating function, corresponding to the operating speed of the object detected by the motion detecting means, to generate the trajectory image expressing motion track of the object and the trajectory image generating means,Image synthesizing means for synthesizing the reference image and the trajectory image to generate a trajectory composite image.
The image processing apparatus according to claim 2 is the image processing apparatus according to claim 1, wherein a user of the image processing apparatus selects an arbitrary frame image from the plurality of frame images stored in the storage unit. The image processing apparatus further includes a reference image selecting unit, wherein the reference image setting unit sets the frame image selected by the reference image selecting unit as the reference image.
[0010]
4. The image processing device according to claim 3, wherein a plurality of frame images are extracted from the moving image information and stored in a storage unit, and a specific frame image is extracted from the plurality of frame images stored in the storage unit. Reference image setting means for selecting a reference image and setting the reference image as a reference image; motion detection means for detecting a feature amount indicating a motion of a subject included in each of the plurality of frame images stored in the storage means; based on the user's instruction, a plurality of different path image generating function selecting means for selecting the trajectory image generation function to be executed from the trajectory image generating function and the trajectory image generating function selected by the path image generating function selector To generate a trajectory image representing a motion trajectory of the subject corresponding to the motion of the subject detected by the motion detection means. Means synthesizes the said reference image and said trajectory image, comprising: the image synthesizing means for generating a trajectory composite image.
An image processing apparatus according to a fourth aspect is the image processing apparatus according to the first, second, or third aspect,An image output means for displaying and outputting the trajectory synthesized image generated by the image synthesizing means is provided.
[0011]
Claims5The described image processing apparatus includes a storage unit that stores moving image information including a plurality of frame images, and a reference image that selects a specific frame image from the plurality of frame images stored in the storage unit and sets the selected frame image as a reference image Setting means, a trajectory image generation function selecting means for selecting a trajectory image generation function to be executed from a plurality of different trajectory image generation functions based on a user's instruction, or an operation speed of a subject; A trajectory image generating unit configured to generate a trajectory image expressing a motion trajectory of a subject included in each of the plurality of frame images using the plurality of frame images stored in the storage unit based on the selection result; Image combining means for combining the reference image and the trajectory image to generate a trajectory combined image,
It is characterized by having.
[0012]
And claims6The image processing method describedAn image processing method executed by an MPU,Extracting a plurality of frame images from the moving image information and storing them in a storage unit, selecting a frame image from the plurality of frame images stored in the storage unit, and setting as a reference image, Detecting a movement of a subject included in each of the stored plurality of frame images; and a trajectory executed from a plurality of different trajectory image generation functions based on a user's instruction or a detected operation speed of the subject. Selecting an image generation function, and processing the plurality of frame images based on the selected trajectory image generation function, the reference image, and the detected motion of the subject to represent a motion trajectory of the subject Generating a trajectory image, and synthesizing the reference image and the trajectory image to generate a trajectory composite image. And wherein the Mukoto.
[0013]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<First embodiment>
FIG. 1 is a block diagram showing a first embodiment of the image processing apparatus according to the present invention.
In FIG. 1, reference numeral 10 denotes a moving image data capturing unit (image information capturing means), reference numeral 20 denotes a frame memory (storage means), reference numeral 30 denotes an MPU, reference numeral 40 denotes a hard disk, reference numeral 50 denotes a monitor such as LCD (image output means), and reference numeral 60 denotes A printer (image output unit), 70 is an input unit such as a key switch (reference image selection unit, locus image selection unit), and 80 is a data / command transmission bus. Here, the MPU 30 includes a motion detection section (motion detection section) 31, a trajectory image generation section (trajectory image generation section) 32, a center image setting section (reference image setting section) 33, and an image synthesis section (image synthesis section). ) 34.
[0014]
These functions are generally as follows.
(1) Video data capturing unit 10
The moving image data capturing unit 10 extracts a plurality of frame images from the moving image data and stores them in a frame memory 20 described later. Here, the moving image data may be constituted by video data or a plurality of continuously shot still images. In short, any frame image may be used as long as it is composed of a plurality of continuous frame images. The frame image to be captured may be an analog image or a digital image.
The schematic configuration of the moving image data capturing unit 10 will be described with reference to FIG.
In FIG. 2, reference numeral 11 denotes a frame image selection unit, 12 denotes an analog-digital converter (hereinafter, referred to as A / D), 13 denotes an image compression unit, and 14 denotes a bus interface.
[0015]
The frame image selecting unit 11 selects frame images at predetermined time intervals from the moving image data according to a command from the MPU 30 described later, and extracts an image signal of each selected frame image. Here, the selected frame image may be all the frame images constituting the moving image data, or may be frame images at specific time intervals, for example, every 1/5 sec or 1/10 sec. It is essential to have a time interval enough to grasp the movement of the subject included in the moving image data. The time interval used for selecting a frame image is determined by a method using a reference value preset by the MPU 30 or a method using an image processing apparatus.User (user)Various methods can be set, such as a method using an arbitrary value instructed via the input unit 70 according to the intention of the user.
When the selected frame image is an analog signal, the A / D 12 converts the analog image signal into a digital image signal.
[0016]
The image compression unit 13 converts the image signal of each frame image converted into a digital image signal by the A / D 12 or extracted as a digital image signal by the frame image selection unit 11 so as to conform to a predetermined image compression standard. To process. The JPEG standard or the like can be applied as a frame image compression encoding method. Here, JPEG (Joint Photographic Coding Experts Group) is a standard for compressing and encoding original image data by a method such as DCT (Discrete Cosine Transform), quantization, and variable-length coding. This is an international standard adopted for still cameras and the like. Note that, as a compression encoding method for image information, since various formats and standards such as GIF, TIFF, LHA, and ZIP are generally used in addition to JPEG, an appropriate method is adopted according to the embodiment. be able to.
The bus interface 14 converts the compressed frame image into a transmission width of the bus 80 and transfers the frame image to the frame memory 20.
[0017]
(2) Frame memory 20
The frame memory 20 is composed of a DRAM (Dynamic Random Access Memory) or the like, is selected by the moving image data capturing unit 10, associates header information such as image identification information with a compression-encoded frame image, and designates it by the MPU 30. Is stored in the stored image data storage area.
FIG. 3 is a conceptual diagram showing an internal area of the frame memory 20.
As shown in FIG. 3, the internal area of the frame memory 20 is roughly divided into a format table area, an information table area, an image data area, and an offset area.
The format table area stores format information that is comprehensive information regarding image information. The information table area includes a header for identifying image information such as image information identification information including number information for identifying image information, and time information indicating a position (time) of moving image data in a time series. Information is stored. The image data area stores compression-encoded frame images (compressed image data), and the offset area stores offset data (blank) for setting the data length of the compressed image data in the image data area to a fixed length. .
As described above, each compressed image data is stored in the image data area in association with the header information stored in the information table area.
[0018]
(3) Motion detection unit 31
The motion detecting section 31 detects a feature amount indicating the motion of the subject from a plurality of frame images stored in the frame memory 20. Here, the feature amount indicating the motion of the subject is information indicating a motion locus or trajectory of the subject, an operation (movement) speed, an operation (movement) direction, and the like, and can be generally expressed as vector information.
As a method of motion detection, frame images at different times are divided into block regions, block matching processing is performed on a reference image for each of the same block regions, and a coordinate position at which an error is minimized between frames of a subject. A block matching method for detecting motion can be applied. It should be noted that the block matching method is widely used in, for example, the IS117172-2 standard internationalized in ISO / IECJTC1 / SC29 / WG11. The block matching method will be described later.
[0019]
(4) Central image setting unit 33
The center image setting unit 33 selects and extracts a specific frame image from a plurality of frame images stored in the frame memory 20, and sets the selected frame image as a center image in image synthesis processing described later. Here, the center image is an image serving as a reference when synthesizing images, and may be an image located at the beginning or end of a series of frame images, or may be a frame image at a specific time. . In short, any method may be used as long as it extracts an image serving as a nucleus (original) to be combined with a trajectory image created by the trajectory image generation unit 32 described later.
[0020]
(5) Locus image generation unit 32
The trajectory image generation unit 32 stores in the frame memory 20 based on the feature amount indicating the motion of the subject detected by the motion detection unit 31 and the subject image included in the center image extracted by the center image setting unit 33. A plurality of frame images are processed to generate a trajectory image representing a motion trajectory or an afterimage according to the motion of the subject.
The trajectory image is generated by determining the subject from a plurality of frame images stored in the frame memory 20 based on the detection result of the motion detection unit 31 and extracting the contour component of the subject from each frame image other than the center image. , A process of multiplexing the extracted contour components, or a process of generating an afterimage region based on the contour components,In addition, the imageA unique trajectory image expressing the movement of the subject is created by performing shading adjustment.
[0021]
(6) Image synthesis unit 34
The image synthesizing section 34 synthesizes the locus image created by the locus image generating section 33 with the central image extracted by the central image setting section 33, and outputs it as a locus composite image.
(7) MPU 30, hard disk 40
The hard disk 40 stores programs executed by the MPU 30 and data necessary for operation. Therefore, the MPU 30 executes the application programs stored in the hard disk 40 to execute the functions of the motion detection unit 31, the trajectory image generation unit 32, the center image setting unit 33, and the image synthesis unit 34 by software. To perform a series of image processing and memory management, which will be described later, and output control to the monitor 50 and the printer 60.
[0022]
(8) Monitor 50, Printer 60
The trajectory composite image synthesized by the image synthesizing unit 34 is displayed or printed out, and is an output device such as a television, a monitor of a personal computer, or a printer. Here, in FIG. 1, the monitor 50 and the printer 60 are shown as being configured to be directly connected to the bus 80. However, the present invention is not limited to this, and a communication interface or the like connected to the bus 80 may be used. It may be a facsimile apparatus, a personal digital assistant (PDA), a personal computer, or the like, which is connected by a communication line via the communication line. In the description of the present invention, in addition to the display output to the monitor 50 and the print output to the printer 60, the entire operation of outputting the locus composite image as a still image is referred to as “display output” for convenience.
(9) Input unit 70
The input unit 70 is various key switches provided in the image processing apparatus, and generates control signals such as execution of an application program by the MPU 30, image processing, and display output to the monitor 50 and the printer 60. Further, it also has a function as a reference image selecting unit or a locus image selecting unit in an embodiment described later. When the present invention is implemented by a personal computer or the like, various input devices such as a keyboard, a mouse, and a pen tablet are included in addition to dedicated key switches provided in the image processing device.
[0023]
Next, a processing operation in the image processing apparatus having the above configuration will be described with reference to the drawings.
FIG. 4 is a flowchart illustrating the processing operation of the image processing apparatus according to the present embodiment. First, the outline of the processing operation will be described with reference to the above-described configuration, and then each step will be individually described.
As shown in FIG. 4, in steps S101 and S102, the moving image data capturing unit 10 selects and extracts a plurality of frame images at predetermined time intervals from the input moving image data so as to conform to a standard such as JPEG. Is subjected to a compression encoding process and stored in a predetermined storage area of the frame memory 20. At this time, header information for identifying the compressed frame image is stored in association with the header information.
ThenStep S103In step S104, the central image setting unit 33 sets a central image serving as a reference in image synthesis from among a plurality of frame images stored in the frame memory 20, and in steps S104 and S105, the motion detecting unit 31 And detects a subject (moving object) accompanying a motion included in the plurality of frame images stored in the frame image, and outputs information indicating a motion characteristic of the subject.
[0024]
ThenStep S106, S107In the trajectory image generation unit 32, the contour components of the subject included in each frame image are extracted based on the above-described motion characteristics of the subject and the center image, and the contour components are multiplexed or Generate an afterimage area based on the component, and further adjust the density of the contour componentSubject to processing,A trajectory image representing the movement of the subject is created.
ThenStep S108In step S109, the image combining unit 34 combines the center image and the trajectory image into one still image, and outputs the obtained still image to the monitor 50, the printer 60, or the like.
As described above, the processing operation of the image processing apparatus according to the present embodiment is roughly divided into a moving image data capturing step, a center image setting step, a motion detecting step, a trajectory image creating step, and an image synthesizing step.
[0025]
Hereinafter, each step will be described with reference to the drawings.
(1) Video data import step
FIG. 5 is a conceptual diagram showing the moving image data capturing step. Hereinafter, a description will be given of a vehicle running as a subject as an example.
Moving image data captured by a video camera or an electronic still camera, or moving image data such as video information of a television broadcast is a set of a series of frame images arranged in time series. , Frame images Fa, Fb, Fc,... At times T2, T6, T10,... Corresponding to the time-series positions of the moving image data VD, for example, at predetermined time intervals from a series of frame images. Selected and extracted by the moving image data capturing unit 10. Note that the selected and extracted frame image may be all of a series of frame images constituting the moving image data VD, and may be one that can grasp the movement of the subject, that is, the running state of the vehicle by the extracted frame image. The time interval is not limited to the above.
The extracted frame images Fa, Fb, Fc,... Are converted into digital image signals in order to easily execute image processing to be described later, and furthermore, in order to effectively use the storage capacity of the frame memory 20, JPEG or the like is used. A predetermined compression encoding process is performed, and the image data is sequentially stored in the image data area in association with image identification information, time information, and the like.
[0026]
(2) Center image setting step
In this step, the center image setting unit 33 selects, for example, the frame image Fb at time T6 from the plurality of frame images Fa, Fb, Fc,... The image is set as the center image by adding header information indicating that the image is an image. As described above, the center image is a reference image at the time of image composition, and the extraction method is to extract an image located at the start end, the end, or an intermediate time among the captured frame images. May be set in advance, or based on a detection result of a motion characteristic of a subject, which will be described later, for example, a vehicle whose traveling vehicle is located at the center position of the frame image may be selected.
[0027]
(3) Motion detection step
In this step, the motion detecting unit 31 detects a traveling vehicle from a plurality of frame images stored in the frame memory 20 and calculates the trajectory or trajectory of the motion.
Hereinafter, a method for detecting the movement of the subject included in the image information will be described with reference to the drawings.
First, block matching will be described as an example of a technique applied to motion detection.
FIG. 6 is a conceptual diagram showing a block matching method applied to the motion detection step.
For example, see “Television Society,“ Image Information Compression ”, Ohmsha, p. 92, 1991], describes a block matching method for detecting the movement of a subject included in moving image data composed of a plurality of continuous frame images.
[0028]
According to the above document, as shown in FIG. 6, a frame image of interest (hereinafter, referred to as a current frame image for convenience) Fn and a frame image at the immediately preceding time (hereinafter, referred to as a previous frame image for convenience). .) Pattern matching is performed on blocks (regions) Bn and Bm at specific positions in Fm. The pattern matching method is, for example, obtaining the absolute value sum S of the difference between the pixel Xn in the block Bn and the pixel Xn-i in the block Bm obtained by shifting the pixel Xn by i pixels according to the following equation (1). The sum of values S, that is, the shift position i that minimizes the evaluation amount is searched to detect the motion amount.
S = Σ | Xn-Xn-i | ... (1)
Here, the sum calculation in the equation (1) is executed for all the pixels Xn belonging to the block Bn.
As described above, in the block matching method, the current frame image is divided into blocks, and a similar position to the previous frame image is searched for by block matching in block units, and the motion speed, direction, and the like of the subject included in the frame image are searched for. Detect motion.
[0029]
As a method applied to motion detection, a method of calculating a motion vector from a change in a characteristic component of a subject region included in a frame image can be applied in addition to the above-described block matching method.
In this method, first, a Laplacian process is performed on a luminance component included in a frame image, a zero-crossing point of the processed image is detected as a region boundary line, and a portion closed by a continuous boundary line is set as an independent region (subject region). ). Alternatively, a color component included in the frame image is analyzed, and a continuous portion having a small change in hue is replaced with a representative color and extracted as one region. Then, a feature component of the extracted region, for example, the position of the center of gravity of the region is detected, and the amount of change between frame images is calculated to obtain a motion vector indicating the amount of parallel movement of the entire region. Therefore, by outputting the calculated change amount as the motion parameter of the region, the value (movement amount, direction) of the motion vector can be obtained by calculation at an arbitrary pixel position in the subject region.
As still another method of motion detection, a method of detecting a large number of contour components in a frame image, grouping the detected large number of contour lines into groups, and detecting the motion of the entire image by neural network processing is applied. You can also.
[0030]
(4) Trajectory image creation step
In this step, the trajectory image generation unit 32 generates a trajectory image representing the running state of the vehicle based on the plurality of frame images stored in the frame memory 20 and the center image.
FIG. 7 is a conceptual diagram illustrating a first example of the trajectory image creation step.
As shown in FIG. 7, based on a plurality of frame images Fa, Fb, and Fc stored in the frame memory 20, based on the movements (movement characteristics) of the vehicles Ca, Cb, and Cc determined in the above-described movement detection step. Trajectory images are generated by extracting only the contour components of and arranging them in chronological order. The trajectory image generated here is different from the mere multi-synthesis of the frame images described in the related art, in that only the outline component is synthesized, so that the trajectory image accurately represents the entire movement of the vehicle. Further, in the center image setting step, a trajectory image is processed based on the position of the vehicle Cb included in the frame image (for example, Fb) set as the center image.
[0031]
More specifically, a plurality of trajectory images are configured using the luminance, saturation, and the like of an outline component (hereinafter, referred to as a reference outline component) included in an image (subject image) of the vehicle Cb included in the center image Fb as a reference value. Of the contour components located near the reference contour component, the brightness and the saturation are approximated to the reference value, and as the distance increases, the brightness and the saturation are reduced to adjust to a so-called blurred state. . At this time, the processing is performed only on the contour components constituting the trajectory image located on the opposite side (rear side; left side in the drawing) of the traveling direction of the vehicle Cb, and is located on the traveling direction side (front side; right side in the drawing). The contour component is deleted. That is, the locus image La is created by performing image processing using the outline component of the frame image belonging to the past than the center image. The above-described trajectory image creation processing is referred to as “first trajectory image creation processing”.
[0032]
Further, as another method, in addition to the above-described multiple composition processing of the contour component, a process of generating and arranging a specific image element expressing the motion direction and the motion state of the subject, for example, a line component extending in the motion direction of the subject, is performed. In addition, as shown in FIG. 7, a locus image Lb may be created. The above-described trajectory image creation processing is referred to as “second trajectory image creation processing”.
In the above-described processing of the trajectory image, a method of extracting and multiplex-synthesizing the contour components of the subject included in the frame image and further performing the processing of each contour component based on the position of the subject included in the center image is described.It has been described,The present invention is not limited to the above-described example as long as it allows a viewer of the synthesized or processed image to accurately recognize the operation state of the subject.
[0033]
Next, a description will be given of a second example of the path image creation step.
In the above-described first example, the description has been given on the assumption that the moving image data is obtained by capturing a state in which the vehicle included in the extracted frame image is traveling with respect to the fixed background. There is a case where moving image data is obtained by fixing a running vehicle at a fixed position in a frame image. The process of creating path image in such a case will be described.
FIG. 8 is a conceptual diagram illustrating a second example of the trajectory image creation step.
As shown in FIG. 8, from a plurality of frame images Fi, Fj, and Fk extracted from the moving image data and stored in the frame memory 20, the background component Gi, Gj, to extract the Gk only. The background components Gi, Gj, and Gk are easily generated by deleting, from the frame images Fi, Fj, and Fk, the subject (vehicles Ci, Cj, and Ck) components whose motion has not been detected in the motion detection step. Then, the extracted background components Gi, Gj, and Gk are arranged in time series to generate a trajectory image. Further, in the center image setting step, the background included in the frame image (for example, Fk) set as the center image Based on a component (hereinafter, referred to as a reference background component) Gk, processing of a locus image is performed.
[0034]
Specifically, the luminance, the saturation, and the like of the reference background component Gk are processed so as to represent the largest degree of blur, and the plurality of background components constituting the trajectory image are arranged near the reference background component Gk. The trajectory image Lc is created by performing a blurring process by approximating the positioned object to the reference value, and adjusting the background component to have the original luminance and saturation as the distance increases.
Note that, similarly to the first example described above, the trajectory image creation processing includes not only the synthesis and processing of the background component, but also the line component expressing the moving (flowing) state of the background component.A combination of the additional processingIt may be the one to use.
[0035]
(5) Image synthesis step
In this step, the image combining unit 34 combines the center image and the trajectory image to generate a still image representing the running state of the vehicle, and outputs the still image via the monitor 50 or the printer 60.
9 and 10 are conceptual diagrams showing a first example of the image synthesizing step. Here, FIG. 9 is a conceptual diagram showing a case in which the trajectory image La created by the above-described first trajectory image creation process and the center image are combined, and FIG. FIG. 7 is a conceptual diagram showing a case where a locus image Lb created by the locus image creation process is combined with a center image.
[0036]
As shown in FIGS. 9 and 10, the frame image Fb set as the center image in the center image setting step and the trajectory images La and Lb created in the trajectory image creation step are combined to perform processing of the vehicle Cb. Still images ED1 and ED2 obtained by synthesizing the trajectory images La and Lb representing the traveling state are obtained. That is, the center image Fb is only a specific one of a series of frame images constituting moving image data, but the trajectory images La and Lb have the outline component and the center component based on the position of the vehicle Cb included in the center image Fb. Since the processing of the area has been performed, the traveling locus of the vehicle Cb is expressed in one frame image Fb by combining the two.
Then, the synthesized still images ED1 and ED2 are transferred to the monitor 50 and the printer 60 via the bus 80 and displayed and output.
[0037]
FIG. 11 is a conceptual diagram illustrating a second example of the image synthesizing step, and illustrates a case where the trajectory image Lc created by the trajectory image creation process illustrated in FIG. 8 and the image of the vehicle Ck are combined.
As shown in FIG. 11, the image component of the vehicle Ck included in the frame image Fk set in the center image in the center image setting step and the trajectory image Lc created in the trajectory image creation step are multiplexed and synthesized. A still image ED3 expressing the running state of the vehicle Ck in the background flow is obtained. That is, as in the first example described above, the center image Fk is only a specific one of a series of frame images constituting moving image data, but the trajectory image Lc includes a background component included in the center image Fk. Since the processing of the contour and the region is performed based on Gk, the traveling state of the vehicle Ck is represented by the flow of the background in one frame image Fk by combining the two.
According to the present embodiment having the above series of steps, by detecting a series of movements of the subject in the moving image data, creating a trajectory image corresponding to the position and movement of the subject, and performing synthesis processing with the center image, A still image in which the movement of the subject is accurately represented can be created. Therefore, since the motion of the moving object can be intuitively grasped with one still image, it can be effectively used as an index of moving image data.
[0038]
<Second embodiment>
Next, a second embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings.
In the present embodiment, in the first embodiment described above, the movement (movement characteristics) of the subject detected by the movement detection unit 31 is analyzed, and the method of creating the locus image is changed according to, for example, the traveling speed of the vehicle. It is like that.
That is, in the image processing apparatus shown in FIG. 1, the trajectory image generation unit 32 analyzes the detection result such as the motion speed and the motion direction of the subject detected by the motion detection unit 31 and changes the trajectory image creation processing. Control function.
Specifically, as shown in FIG. 9, in the still image ED1 in which the locus image La created by the first locus image creation process is synthesized with the center image Fb, the movement of the vehicle Cb is recognized to be relatively slow. On the other hand, as shown in FIG. 10, in the still image ED2 obtained by synthesizing the locus image Lb created by the second locus image creating process with the center image Fb, the movement of the vehicle Cb is It is expressed as a relatively faster recognition. Therefore, based on such characteristics, the operation speed of the subject detected by the motion detection unit 31 is determined, for example, by comparing it with a predetermined threshold value to determine whether the operation is a low-speed operation or a high-speed operation. switching the first and second path image creation processing as appropriate.
[0039]
Here, the relationship between the movement speed of the subject and the trajectory will be described. When the subject is moving at a low speed, the contour of the moving object is grasped relatively clearly, which corresponds to the human visual (retinal) characteristic. Then, by performing multiple synthesis for the number of frame images without blurring the outline of the subject (moving object) included in each frame image, a low-speed operation state can be expressed, and the motion of the subject can be appropriately recognized. . In the present invention, in addition to the multiple synthesis of the contour components, the brightness and saturation of the contour near the subject included in the center image are approximated to those of the center image so that the line segments and colors are clearly displayed. In a contour far from the subject, processing is performed to blur the line segments and colors, and a processing process for expressing the movement of the subject more realistically is performed.
On the other hand, when the subject moves at a high speed, the outline of the moving object is grasped extremely vaguely. Thus, a high-speed operation state can be expressed, and the movement of the subject can be appropriately recognized.
[0040]
Hereinafter, the processing operation of the present embodiment will be described with reference to the flowchart of FIG. Steps equivalent to those in the above-described first embodiment will be simplified.
As shown in FIG. 12, in steps S201 and S202, a plurality of frame images are selected and extracted from the moving image data at predetermined time intervals, compression-encoded, and stored in a predetermined storage area of the frame memory 20. ThenStep S203In step S204, a center image is set from a plurality of frame images stored in the frame memory 20, and in step S204, a subject (moving object) included in the plurality of frame images is detected by applying the above-described block matching method or the like. At the same time, the motion characteristics such as the motion speed and direction of the subject are calculated.
In steps S205 to S207, the calculated operation speed is compared with a preset threshold speed. If the calculated operation speed is lower than the threshold, the subject is a low-speed moving object. Is determined, the above-described first trajectory image creation processing is selected, and processing is performed so that the contour components forming the trajectory image are relatively clearly expressed. On the other hand, when the calculated operation speed is higher than the threshold value, it is determined that the subject is a high-speed moving object, and the above-described second trajectory image creation processing is selected, and in addition to the multiple synthesis of the contour components, , A processing for superimposing a line component or the like expressing a motion direction or the like is performed.
ThenStep S208, S209In step (1), the center image and the trajectory image are combined to form one still image, which is displayed and output via the monitor 50 and the printer 60.
[0041]
According to the present embodiment having the above series of steps, a method of creating a locus image based on a series of detected movements of a subject, particularly an operating speed (moving speed)(Function to execute the locus image creation process; locus image generation function)Can be changed to execute a trajectory image creation process of creating a trajectory image corresponding to the operation speed, so that a still image that can intuitively recognize the movement of the subject can be created.
In the present embodiment, a case where the first and second trajectory image creation processes are used as the processing method selected according to the operation speed of the subject is described.It has been shown,The method for processing the background component shown in FIGS. 8 and 11 may be changed. In short, any method other than the above method may be used as long as it allows a viewer of the synthesized still image to accurately recognize the operation state of the subject. Also, as the processing for determining the operation speed of the subject, a case has been described where comparison processing with one preset threshold value is performed. However, multi-level determination based on a plurality of threshold values may be performed. The luminance and the saturation may be weighted using the speed as it is.
[0042]
<Third embodiment>
Next, a third embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings.
In the present embodiment, selection and setting processing of the center image in the center image setting step is changed according to a command input from the input unit 70 or the like.
That is, in the image processing apparatus shown in FIG. 1, the central image setting unit 33 sends the central image setting unit 33 to the frame memory 20 in accordance with the central image selection command input from the input unit 70 or via a communication line (not shown). It has a function of extracting an arbitrary frame image from a plurality of stored frame images and setting the extracted frame image as a center image.
Specifically, a plurality of frame images stored in the frame memory 20 are displayed and output via the monitor 50 and the printer 60, and an input of a command for specifying a frame image to be selected as the center image is input to the user of the image processing apparatus. Prompt. The user browses the displayed and output frame images, selects an arbitrary frame image from the displayed frame images, inputs a center image selection command via an input unit 70 such as a key switch or a keyboard, and inputs the center image selection instruction to the frame memory 20. Header information indicating the center image is added to the information table area of the image.
[0043]
Therefore, the user can easily input the frame image that seems to be most appropriate for expressing the motion state of the subject or the frame image that is most appropriate for producing the special effect from the input unit 70 or the like, and can easily input the frame image. it can be set as the center image.
Hereinafter, the processing operation of the present embodiment will be described with reference to the flowchart of FIG. Steps equivalent to those in the above-described first or second embodiment will be simplified.
[0044]
As shown in FIG. 13, in steps S301 and S302, a plurality of frame images are selected and extracted from the moving image data at predetermined time intervals, subjected to compression encoding processing, and stored in a predetermined storage area of the frame memory 20. ThenStep S303, All or some of the plurality of frame images stored in the frame memory 20 are displayed and output via the monitor 50, the printer 60, etc., and the user is prompted to select a frame image to be set as the center image and to input a command. .
Here, the display output method of the frame images by the monitor 50 and the printer 60 may be a method of sequentially outputting the frame images in time series, or a method of outputting a plurality of frame images collectively on the divided screen. There may be. Further, the frame images to be displayed and output may be all the frame images stored in the frame memory 20, or may be some frame images. That is, in the frame images Fa, Fb, and Fc extracted in FIG. 5, all of these frame images may be displayed and output, or only some of the odd-numbered frame images Fa and Fc may be displayed and output. May be used. In short, what is necessary is just to display and output a frame image that can be used as a center image in the image synthesis processing.
[0045]
Next, in step S304, header information is set so that the frame image selected and designated by the user becomes the center image. Further, in step S305, the subject (moving object) included in the plurality of frame images is detected by applying the above-described block matching method or the like, and the operation state of the subject is output. Then, in step S306, a predetermined trajectory image is generated based on the operation state and the center image of the subject,Steps S307 and S308In step (1), the center image and the trajectory image are combined to form one still image, which is displayed and output via the monitor 50 and the printer 60.
According to the present embodiment having the above series of steps, the user does not univocally select and set the center image from among the plurality of frame images, but browses the displayed and output frame image and allows the user to freely select the center image. Since the frame image can be designated and set, it is possible to realize more various image expressions while accurately expressing the movement of the subject. Note that the trajectory image creation step is not limited to the method described in the first embodiment, but may be a method in which the processing method is appropriately changed according to the motion characteristics of the subject as described in the second embodiment. Good.
[0046]
<Fourth embodiment>
Next, a fourth embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings.
In the present embodiment, the change of the trajectory image creation processing in the above-described second embodiment is not controlled based on the motion characteristics of the subject, but via the input unit 70 or the like according to the user of the image processing apparatus. it is obtained to be able to arbitrarily set.
That is, in the image processing apparatus shown in FIG. 1, the trajectory image generation unit 32 outputs a trajectory image according to a trajectory image creation process selection command input from the input unit 70 or via a communication line (not shown). It has a function of arbitrarily selecting a method of creating. The process of creating a trajectory image to be selected is a process of creating a trajectory image according to the motion characteristics of the subject, that is, the first and second trajectory image creation processes described above.And, background componentIn addition to the above-described processing, it is also possible to arbitrarily select a process of creating a special trajectory image that emphasizes a subject included in the center image or characterizes the subject.
[0047]
Here, an example of a process of creating and synthesizing a special trajectory image that emphasizes or characterizes a subject will be described with reference to the drawings.
FIG. 14 is a conceptual diagram illustrating an example of a special trajectory image creation process, and FIG. 15 is a conceptual diagram illustrating an example of a combining process of a special trajectory image and a center image.
As shown in FIG. 14, only a contour component of the vehicle is extracted from a plurality of frame images Fa to Fe stored in the frame memory 20 based on the movement of the vehicles Ca to Ce determined in the motion detection step, and (Multiple composition), a trajectory image expressing the entire movement of the vehicle is generated. Then, based on the position and the contour components of the vehicle Cc included in the frame image (for example, Fc) set as the center image, the contour components located in front of and behind the vehicle Cc are blurred as the distance from the vehicle Cc increases. The locus image LQ is created by performing processing to generate the locus image.
[0048]
Next, as shown in FIG. 15, by combining the frame image Fc set as the center image and the special trajectory image LQ, the vehicle Cc is emphasized in the entire motion trajectories of the vehicles Ca to Ce. special still image EDQ is obtained.
Accordingly, the user can easily set a desired still image by inputting and setting, for example, a method of creating a locus image that seems to be most appropriate for producing a special effect via the input unit 70 such as a key switch or a keyboard. Can be created.
Hereinafter, the processing operation of the present embodiment will be described with reference to the flowchart of FIG. Steps equivalent to those in the above-described first to third embodiments will be briefly described.
[0049]
As shown in FIG. 16, in steps S401 and S402, a plurality of frame images are selected and extracted from the moving image data at predetermined time intervals, subjected to compression encoding processing, and stored in a predetermined storage area of the frame memory 20. ThenStep S403In step S404, a center image is set from among a plurality of frame images stored in the frame memory 20, and in step S404, a subject (moving object) included in the plurality of frame images is detected by applying the above-described block matching method or the like. as well as to output the operation state of the subject.
Then, in steps S405 and S406, a predetermined trajectory image is generated based on the trajectory image creation method selected and set by the user,Steps S407 and S408In step (1), the center image and the trajectory image are combined to form one still image, which is displayed and output via the monitor 50 and the printer 60.
According to the present embodiment having the above series of steps, since the method of creating a locus image can be arbitrarily selected and set according to the user's will, a special locus image created using the motion locus of the subject Is synthesized with the center image, a still image exhibiting a special effect can be easily created.
It should be noted that the image processing apparatus of the present invention can be favorably applied to image processing apparatuses such as video players, facsimile apparatuses, and printers in addition to the above-described video cameras, electronic still cameras, and personal computers. Nor.
[0050]
【The invention's effect】
Claim1, 3, 5 or 6According to the described invention, since a trajectory image centering on the subject included in the frame image set as the reference image and according to the motion of the subject is generated, the intuition of the motion of the subject captured as the moving image information is intuitive. Still images that can be visually recognized can be easily created and output. In particular, a method of generating a trajectory image to be executed is selected based on at least the movement of the subject from among a plurality of different trajectory image generating methods (trajectory image generating function), and the contour components of the subject are multiplexed and synthesized. A contour image near the subject of the image is clearly displayed, and a still image (trajectory composite image) that accurately reflects the operation state of the subject is created by processing so that blurring or blurring occurs as the distance from the subject increases. be able to.
And claims4By having the image output means according to the described invention, the still image can be displayed and output on a television monitor or a monitor of a personal computer, or printed out by a printer or the like. Therefore, the moving image information can be expressed as a still image by a simple method and in various expression formats, and can be effectively used for purposes such as index information of the moving image information.
[0051]
In particular, claim 1According to the invention described above, the motion speed of the subject included in the frame image extracted from the moving image information is detected, and the method of generating the locus image (trajectory image generation function) is changed in accordance with the detection result. it is possible to generate the trajectory image corresponding to the operating speed of the. Therefore, since the sense of speed expressed by the locus image varies depending on the motion speed of the subject, the entire motion of the subject and the motion speed can be intuitively grasped only by looking at the created still image.
Claims2According to the described invention, a frame image serving as a reference image can be arbitrarily selected and set by a user (user) of the image processing apparatus, and a subject included in the reference image is centered, In addition, since a trajectory image is generated in accordance with the movement of the subject, a still image in which the subject located at an arbitrary position is emphasized can be easily created, and the movement of the subject can be accurately determined using a wider variety of expression formats. it is possible to grasp.
Claims3According to the described invention, a method of generating a trajectory image can be arbitrarily selected and set by a user (user) of the image processing apparatus, so that a wider variety of expression formats can be realized. , it is possible to create a still image with special effects easily. In particular, a trajectory image desired by the user can be generated by providing, as a method of generating a trajectory image, blurring of a contour component, gradation processing, addition of a specific image expressing speed, and the like.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a first embodiment of an image processing apparatus according to the present invention.
FIG. 2 is a block diagram illustrating a schematic configuration of a moving image data capturing unit.
FIG. 3 is a conceptual diagram showing an internal area of a frame memory.
FIG. 4 is a flowchart illustrating a processing operation of the image processing apparatus according to the first embodiment.
FIG. 5 is a conceptual diagram showing a moving image data capturing step.
FIG. 6 is a conceptual diagram illustrating a block matching method applied to a motion detection step.
FIG. 7 is a conceptual diagram showing a first example of a trajectory image creation step.
FIG. 8 is a conceptual diagram showing a second example of the trajectory image creating step.
FIG. 9 is a conceptual diagram showing a case where a locus image created by a first locus image creation process and a center image are combined.
FIG. 10 is a conceptual diagram showing a case where a locus image created by a second locus image creation process and a center image are combined.
FIG. 11 is a conceptual diagram showing a second example of the image synthesizing step.
FIG. 12 is a flowchart illustrating a processing operation of the image processing apparatus according to the second embodiment.
FIG. 13 is a flowchart illustrating a processing operation of the image processing apparatus according to the third embodiment.
FIG. 14 is a conceptual diagram illustrating an example of a special trajectory image creation process applied to the fourth embodiment.
FIG. 15 is a conceptual diagram showing an example of a synthesis process of a special trajectory image and a center image applied to the fourth embodiment.
FIG. 16 is a flowchart illustrating a processing operation of the image processing apparatus according to the fourth embodiment.
FIG. 17 is a diagram showing the concept of a method of extracting and displaying one image from a plurality of frame images constituting moving image information.
FIG. 18 is a diagram showing the concept of a method of extracting a plurality of frames from a plurality of frame images constituting moving image information and collectively displaying them on a divided screen.
FIG. 19 is a conceptual diagram showing a conventional method of expressing moving image information as a still image.
[Explanation of symbols]
10 video data capturing unit (image information capturing means)
20 frame memory (storage means)
30 MPU
31 motion detection unit (motion detection means)
32 path image generating unit (trace image producing means)
33 center image setting unit (reference image setting means)
34 Image Synthesizing Unit (Image Synthesizing Means)
40 Hard Disk
50 monitor (image output means)
60 Printer (image output means)
70 input unit (reference image selecting means, the trajectory image selection means)
80 bus

Claims (6)

An image information capturing unit that extracts a plurality of frame images from the moving image information and stores the extracted frame image in a storage unit;
A reference image setting unit that selects a specific frame image from the plurality of frame images stored in the storage unit and sets it as a reference image;
Motion detection means for detecting the motion of the subject included in each of the plurality of frame images stored in the storage means,
A trajectory image generation function selection unit that selects a trajectory image generation function to be executed from a plurality of different trajectory image generation functions based on the operation speed of the subject detected by the motion operation detection unit;
The trajectory image generation function selected by the trajectory image generation function selection means is executed to generate a trajectory image representing the motion trajectory of the subject corresponding to the motion speed of the subject detected by the motion detection means Trajectory image generating means,
Image combining means for combining the reference image and the trajectory image to generate a trajectory combined image,
An image processing apparatus comprising: The image processing apparatus further includes a reference image selection unit for selecting an arbitrary frame image from the plurality of frame images stored in the storage unit, and the reference image setting unit includes the reference image selection unit. The image processing apparatus according to claim 1, wherein the frame image selected by the following is set as the reference image. Extracting a plurality of frame images from the moving image information, an image information acquisition means for storing in the storage means,
A reference image setting means for selecting a specific frame image from the stored plurality of frame images in the storage means is set as a reference image,
Motion detection means for detecting a movement of an object included in each of the stored plurality of frame images in the storage means,
A trajectory image generation function selecting unit that selects a trajectory image generation function to be executed from a plurality of different trajectory image generation functions based on an instruction of a user of the image processing apparatus;
The trajectory image generation function selected by the trajectory image generation function selection means is executed to generate a trajectory image representing the motion trajectory of the subject corresponding to the motion of the subject detected by the motion detection means. Trajectory image generation means,
By combining said trajectory image and the reference image, an image synthesizing means for generating a trajectory composite image,
An image processing apparatus comprising: The image processing apparatus according to claim 1, 2, 3, wherein the has an image output means for displaying and outputting the locus-synthesized image generated by the image synthesizing unit. Storage means for storing moving image information composed of a plurality of frame images;
A reference image setting unit that selects a specific frame image from the plurality of frame images stored in the storage unit and sets it as a reference image;
Trajectory image generation function selection means for selecting a trajectory image generation function to be executed from a plurality of different trajectory image generation functions based on a user's instruction or an operation speed of a subject,
Based on the selection result of the trajectory image generation function selection means, using the plurality of frame images stored in the storage means, to generate a trajectory image expressing the motion trajectory of the subject included in each of the plurality of frame images Trajectory image generating means for generating,
Image combining means for combining the reference image and the trajectory image to generate a trajectory combined image,
An image processing apparatus comprising: An image processing method executed by an MPU,
Extracting a plurality of frame images from the moving image information and storing the extracted frame images in a storage unit;
Selecting a frame image from the plurality of frame images stored in the storage means, and setting the frame image as a reference image;
Detecting a motion of a subject included in each of the plurality of frame images stored in the storage unit;
A step of selecting a trajectory image generation function to be executed from a plurality of different trajectory image generation functions based on a user's instruction or a detected operation speed of the subject,
Processing the plurality of frame images based on the selected trajectory image generation function, the reference image, and the detected movement of the subject, and generating a trajectory image expressing a motion trajectory of the subject;
Composing the reference image and the trajectory image to generate a trajectory composite image;
An image processing method comprising:

Images