JP4240098B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method for displaying and outputting moving image information captured by a video camera, an electronic still camera, or the like as a still image. The present invention relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method for displaying and outputting moving image information captured by a video camera, an electronic still camera, or the like as a still image.

  In recent years, the spread of AV equipment and the improvement of functions have been remarkable. In particular, highly functional image recording devices such as video cameras and electronic still cameras have come to be used in various situations including consumer and business use. Images taken by these devices are simply displayed and output on a monitor of a liquid crystal panel mounted on the device main body, a television or a personal computer (hereinafter abbreviated as a personal computer) connected via a cable or the like. It is also possible to print out directly via a printer or the like. Conventionally, moving image information is configured as a method of expressing general moving image information (including video information of a television broadcast) as a still image, not limited to moving images taken by a video camera, an electronic still camera, or the like. A method of extracting an arbitrary frame image from a plurality of frame images and displaying it alone, and displaying a plurality of frame images sequentially or collectively on a divided screen is known.

The prior art will be specifically described below.
FIG. 13 is a conceptual diagram of a method for selecting and displaying an arbitrary frame image from a plurality of frame images constituting moving image information. In FIG. 13, it is assumed that the moving image data VD includes an image in which the vehicle X travels from the left to the right in the drawing. Here, the moving image data VD can be considered to be composed of a plurality of frame images (still images) arranged in time series. Accordingly, the user of the image processing apparatus once reproduces and browses the moving image data VD, grasps the movement of the subject (vehicle), the change in the shooting state, and the like, and then selects an arbitrary frame image from the series of frame images. By instructing the selection, an image of an arbitrary scene included in the moving image data can be displayed and output as a still image. For example, as shown in FIG. 13, the switches of the image processing apparatus are operated so as to select a desired frame image (here, the frame image F4 at a specific time T4) from a series of frame images constituting the moving image data. By doing so, one arbitrary scene in which the vehicle X travels can be extracted from the moving image data, and can be displayed and output on a monitor or the like. When there are a plurality of frame images for which selection has been instructed, for example, they are displayed and output in time series on the monitor in the scroll display mode or collectively in the multi-screen mode.

  In the method for selecting and displaying still images as described above, it is possible to recognize the photographing contents such as a change in the photographing situation and a series of movements of the subject only after reproducing and displaying all of the moving image data. The user of the processing device must once reproduce and browse the moving image data VD, grasp the change in the shooting situation, the movement of the subject, etc., and select and instruct the frame image including the desired scene. There are problems that the confirmation work takes a long time, and that various editing work (operations) are required when outputting desired photographing contents as a still image, which is extremely complicated.

  Therefore, the present invention automatically extracts still images according to changes in shooting conditions included in a series of moving image data, movements of subjects, and the like, and displays and outputs them in an image representation that enables intuitive recognition of shooting contents. It is an object of the present invention to provide an image processing apparatus and an image processing method that can be used.

The image processing apparatus according to claim 1, wherein a feature amount detection unit that detects a change in a feature amount included in a plurality of frame images constituting the moving image information, and a change in the feature amount detected by the feature amount detection unit. A frame image selection unit that selects a specific frame image from the plurality of frame images, and a frame image enhancement that performs an enhancement process to change a display frame shape on the specific frame image selected by the frame image selection unit And image output means for displaying and outputting as a still image the plurality of frame images including the specific frame image that has undergone enhancement processing for changing a display frame shape by the frame image enhancement means. To do.
The image processing apparatus according to claim 2 , wherein a feature amount detection unit that detects a change in a feature amount included in a plurality of frame images constituting moving image information, and a change in the feature amount detected by the feature amount detection unit. A frame image selection unit that selects a specific frame image from the plurality of frame images, a frame image enhancement unit that performs a predetermined enhancement process on the specific frame image selected by the frame image selection unit, Print output means for printing out the plurality of frame images including the specific frame image enhanced by the frame image enhancement means as still images.

The image processing device according to claim 4 is the image processing device according to any one of claims 1 to 3, wherein the feature amount is movement of a subject included in a plurality of frame images constituting the moving image information. It is characterized by.
The image processing device according to claim 5 is the image processing device according to any one of claims 1 to 3, wherein the feature amount is audio information accompanying each of a plurality of frame images constituting the moving image information. It is characterized by that.
The image processing apparatus according to claim 6, wherein a moving image information input unit that inputs moving image information including a plurality of frame images, and a moving image that extracts frame images at predetermined time intervals from the moving image information input by the moving image information input unit. An information extraction unit; a feature amount detection unit that detects a change in a feature amount included in a plurality of frame images extracted by the moving image information extraction unit; and a change in the feature amount detected by the feature amount detection unit. Frame image selecting means for selecting a specific frame image from the plurality of frame images, frame image enhancing means for applying a predetermined enhancement process to the specific frame image selected by the frame image selecting means, and the frame The plurality of frame images including the specific frame image enhanced by the image enhancement means are displayed as still images. An image output means for, characterized by comprising a.

The image processing method according to claim 7 , wherein a feature amount detection unit detects a change in a feature amount included in a plurality of frame images constituting moving image information, and the feature amount detected by the feature amount detection unit . Based on the change, a step of causing the display image selection unit to select a specific frame image from the plurality of frame images, and an enhancement process for changing a display frame shape to the specific frame image selected by the display image selection unit Including causing the image processing unit to execute processing, and causing the image output unit to display and output the plurality of frame images including the specific frame image subjected to the enhancement processing by the image processing unit as still images. It is characterized by.
The image processing method according to claim 8, wherein a feature amount detection unit detects a change in a feature amount included in a plurality of frame images constituting moving image information, and the feature amount detected by the feature amount detection unit. Based on the change, image processing includes a step of causing the display image selection unit to select a specific frame image from the plurality of frame images, and a process of performing a predetermined enhancement process on the specific frame image selected by the display image selection unit And a step of causing the printer to print out the plurality of frame images including the specific frame image emphasized by the image processing unit as still images.

According to the present invention, the frame image selected based on the change in the feature amount can be displayed and output with a special image representation different from the normal image display. Therefore, even when a series of frame images are displayed, the image is captured. A frame image indicating a change in the situation or the like can be highlighted, and the user of the image processing apparatus can be appropriately recognized.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<First Embodiment>
FIG. 1 is a block diagram showing a first embodiment of an image processing apparatus according to the present invention.
In FIG. 1, 10 is a moving image data capturing unit, 20 is a frame memory, 30 is an MPU, 40 is a hard disk, 50 is a monitor such as an LCD (image output means), 60 is a printer (image output means), and 70 is a key switch. , Etc. (feature amount selection means, processing method selection means), 80 is a data / command transmission bus. Here, the MPU 30 includes functions of a feature amount detection unit (feature amount detection unit) 31, a display image selection unit (frame image selection unit) 32, and an image processing unit (frame image processing unit) 33. Has been.

Each of these functions is generally as follows.
(1) Moving image data capturing unit 10
The moving image data capturing unit 10 extracts a plurality of frame images from the moving image data together with audio information and stores them in a frame memory 20 described later. Here, the moving image data may be composed of video data or a plurality of still images taken continuously, or may be composed only of image information without including audio information. . In short, in the present invention, the moving image data only needs to be composed of a sequence of at least a plurality of frame images, and the captured frame image may be an analog image or a digital image.
A schematic configuration of the moving image data capturing unit 10 will be described with reference to the drawings. Here, it is assumed that the moving image data has a data configuration in which audio information is attached to image information. Therefore, when moving image data composed only of image information is targeted, a configuration for signal processing of audio information is not included.

In FIG. 2, 11 is a frame image selection unit, 12 is an image / sound separation unit, 13a is an analog-digital converter for image signals (hereinafter referred to as image A / D), and 13b is an analog-digital conversion for audio signals. 14a is an image signal compression unit, 14b is an audio signal compression unit, 15 is an image / sound synthesis unit (hereinafter referred to as a mixer), and 16 is a bus interface. is there.
The frame image selection unit 11 selects and extracts a frame image together with audio information from the moving image data at a predetermined time interval in accordance with a command from the MPU 30 described later. The selected frame image may be all frame images constituting the moving image data, or may be a frame image at a specific time interval, for example, 1/5 sec or 1/10 sec. It is essential to have a time interval that can detect a change in the feature amount included in the data. The definition of the feature amount will be described later. The time interval used when selecting a frame image is a method using a reference value preset by the MPU 30 or an arbitrary value instructed via the input unit 70 by the user of the image processing apparatus. Various methods such as a method can be set.

The image / sound separation unit 12 separates the image signal and the sound signal by using the difference between the frequency bands of the image and the sound, and executes the subsequent signal processing through separate paths of the image and the sound.
The image A / D 13a converts the analog image signal into a digital image signal when the selected frame image is an analog signal. Further, the audio A / D 13b converts an analog audio signal into a digital audio signal when the audio information accompanying the selected frame image is an analog signal.

  The image signal compression unit 14a complies with a predetermined image compression standard for the image signal of each frame image converted into a digital image signal by the image A / D 13a or extracted as a digital image signal by the frame image selection unit 11. To process. The JPEG standard or the like can be applied as a compression encoding method for frame images. Here, JPEG (Joint Photographic Coding Experts Group) is a standard that compresses and encodes original image data by DCT (Discrete Cosine Transform), quantization, variable length coding, and the like. It is an international standard adopted for still cameras. Note that, as a compression encoding method of image information, various formats and standards such as GIF, TIFF, LHA, ZIP, etc. are generally used in addition to JPEG, and therefore an appropriate method is adopted according to the embodiment. be able to.

  The audio signal compression unit 14b converts the audio signal attached to each frame image that has been converted into a digital audio signal by the audio A / D 13b or extracted as a digital audio signal by the frame image selection unit 11 into a predetermined audio signal. Process to comply with compression standards. An MPEG standard or the like can be applied as a compression encoding method for audio signals. Here, MPEG (Motion Picture Coding Experts Group) is a prediction code that compresses and encodes motion-compensated difference data between images in a moving image composed of a plurality of images (screens) continuous in time series. However, it can also be applied to compression coding of audio signals.

As described above, if the image signal is compressed and encoded according to the JPEG standard and the audio signal is encoded according to the MPEG standard, the JPEG standard and the MPEG standard installed in the popular personal computer are supported with the spread of the Internet in recent years. Image processing can be easily realized by software.
The mixer 15 associates the image signal and the audio signal that are compression-encoded according to a predetermined standard, and synthesizes them into one frame data (packetization), while the bus interface 16 converts the synthesized frame data into the frame data. The data is converted into the transmission width of the bus 80 and transferred to the frame memory 20.

(2) Frame memory 20
The frame memory 20 is constituted by a DRAM (Dynamic Random Access Memory) or the like, and associates header information with the image signal and the audio signal (frame data) selected and compression-encoded by the moving image data capturing unit 10, and the MPU 30. The image data is stored in the image data storage area designated by.
FIG. 3 is a conceptual diagram showing an internal area of the frame memory 20.
As shown in FIG. 3, the internal area of the frame memory 20 is roughly divided into a format table area, an information table area, a data area, and an offset area.
The format table area stores format information that is comprehensive information related to image information. The information table area includes headers for identifying image information such as image information identification information including number information for identifying image information, time information indicating a position (time) of moving image data on a time series. Information is stored. The data area stores frame data, that is, the compression-encoded image signal and the accompanying audio signal together, and the offset area is offset data for fixing the data length of the frame data in the data area. (Blank) is stored.
In this way, each frame data is stored in the data area in association with the header information stored in the information table area.

(3) Feature amount detection unit 31
The feature quantity detection unit 31 detects a change in the feature quantity of the image signal or the audio signal from a plurality of frame data stored in the frame memory 20. Here, the feature amount of the image signal is an image of the frame data, that is, a subject (moving object) included in the frame image, the brightness of the screen, saturation, and the like, and the change of the image can be appropriately extracted. Is essential. The feature amount of the audio signal is the volume (level) of the sound accompanying the image, the sound range (frequency band), and the like, and it is essential that the change in the sound can be appropriately extracted. . That is, continuous frame data is monitored for a predetermined change in the feature amount, that is, a drastic change or continuity (non-change state) exceeding a preset threshold, The change characteristics of images and sounds included in frame data (moving image data) are grasped.
Note that a feature amount detection method will be described later.

(4) Display image selection unit 32
The display image selection unit 32 selects specific frame data from the plurality of frame data stored in the frame memory 20 based on the change characteristics of the image or sound included in the frame data detected by the feature amount detection unit 31. Selection, extraction, and display display output to the monitor 50 or the printer 60 via the image processing unit 33 described later or directly. Here, selection and extraction of specific frame data is to select frame data including an image immediately after switching of a shooting situation or immediately after a sudden movement of a subject, which is clarified by the above-described change characteristics of an image or sound. Alternatively, the frame data immediately before and after the switching or operation may be selected. In short, it is only necessary to select and extract an image that allows the user of the image processing apparatus to recognize that a change in the feature amount has occurred when the display is output via the monitor 50 or the printer 60 described later. .

(5) Image processing unit 33
The image processing unit 33 processes the image included in the frame data selected and extracted by the display image selection unit 32 into a predetermined expression format.
Processing to a predetermined expression format includes enlargement of the image size to be displayed and output, change of display position and display image quality, change of tone, deformation of image frame (same size transformation, non-same size transformation, etc.), etc. This means a change to a highlighted format in comparison with other images displayed before or after the image.

(6) MPU 30 and hard disk 40
The hard disk 40 stores programs executed by the MPU 30 and data necessary for operation. Therefore, the MPU 30 executes the application program stored in the hard disk 40 to realize the functions of the above-described feature amount detection unit 31, display image selection unit 32, and image processing unit 33 in software, A series of image processing, memory management, and output control to the monitor 50 and the printer 60 which will be described later are performed.
(7) Monitor 50, printer 60
An image selected by the display image selection unit 32 or an image processed by the image processing unit 33 is displayed or printed out, and is an output device such as a monitor of a television or a personal computer or a printer. Here, in FIG. 1, the monitor 50 and the printer 60 are shown as being directly connected to the bus 80, but the present invention is not limited to this, and a communication interface connected to the bus 80, etc. It may be a facsimile machine, a personal digital assistant (PDA), a personal computer or the like connected via a communication line. In the description of the present invention, in addition to the display output to the monitor 50 and the print output to the printer 60, the entire operation of outputting an image is described as “display output” for convenience.

(8) Input unit 70
The input unit 70 is various key switches provided in the image processing apparatus, and generates control signals such as execution of application programs by the MPU 30, image processing, and display output to the monitor 50 and the printer 60. In addition, it also has a function as a selection unit for a feature amount and an image processing method in an embodiment described later. In addition to dedicated key switches provided in the image processing apparatus, various input devices such as a keyboard, a mouse, and a pen tablet are included when the present invention is implemented by a personal computer or the like.
In the present embodiment, a configuration has been described in which audio data is preliminarily attached and synthesized as image data as moving image data. However, the present invention is not limited to this embodiment, and video Each piece of information may be captured and processed as separate signals, such as image information and audio information immediately after being captured and recorded by a camera, an electronic still camera, or the like. In this case, when the captured information is an analog signal, the image signal compression unit 14a and the audio signal compression are directly performed via the image A / D 13a and the audio A / D 13b. A configuration input to the unit 14b can be applied.

Next, processing operations in the image processing apparatus having the above-described configuration will be described with reference to the drawings.
FIG. 4 is a flowchart showing the processing operation of the image processing apparatus according to this embodiment. First, the outline of the processing operation will be described with reference to the configuration described above, and then each step will be described individually.
As shown in FIG. 4, in steps S101 and S102, the moving image data capturing unit 10 selects and extracts a series of frame images and accompanying audio information from the input moving image data at predetermined time intervals. Then, the data is compressed and encoded so as to comply with a standard such as MPEG and stored in a predetermined storage area of the frame memory 20. At this time, the compression-coded image signal and audio signal are associated with each other, synthesized into one frame data, and stored in association with predetermined header information.
Next, in steps S103 and S104, the feature amount detection unit 31 detects feature amounts included in a series of frame data stored in the frame memory, and grasps change characteristics of images and sounds.

Next, in step S105, the display image selection unit 32 discriminates switching of shooting conditions, a sudden movement of the subject, and the like based on the above-described change characteristics of the image and sound, and for example, a frame including the image immediately after the change. Select data. Next, in step S106, the image processing unit 33 performs processing such as increasing the size of the image to be displayed and outputting, changing the display position and display image quality, changing the tone of the image, and deforming the image frame. In step S107, the display image created by the image processing unit 33 is displayed and output on the monitor 50, the printer 60, or the like.
As described above, the processing operation of the image processing apparatus in the present embodiment is roughly divided into a moving image data capturing step, a feature amount detection step, a display image selection step, and an image processing step.

Hereinafter, each step will be described with reference to the drawings.
(1) Movie Data Capture Step FIG. 5 is a conceptual diagram showing a movie data capture step. Hereinafter, the movements of the vehicle X traveling as a subject and the stopped vehicle Y will be described as examples.
Moving image data shot by a video camera or electronic still camera, or moving image data such as video information of a television broadcast, is a set of a series of frame images arranged in a time series and synthesized data of audio information accompanying them. Therefore, in this step, as shown in FIG. 5, at a predetermined time interval from a series of frame images, for example, times T0, T2, T4, T6 corresponding to positions on the time series of the moving image data VD, The frame images F0, F2, F4, F6, F8, F10,... Of T8, T10,... Are selected by the moving image data capturing unit 10 together with accompanying audio data A0, A2, A4, A6, A8, A10,. Shall be extracted. Note that the frame image selected and extracted may be all of a series of frame images constituting the moving image data VD, that is, a change in the shooting situation and the movement of the subject can be grasped by the extracted frame image and sound. If it is a thing, it will not be limited to the time interval mentioned above.
The extracted frame images F0, F2, F4, F6, F8, F10,... And the accompanying audio data A0, A2, A4, A6, A8, A10,. In order to make effective use of the storage capacity of the frame memory 20, it is converted into an image signal and a digital audio signal, and is subjected to a predetermined compression encoding process such as JPEG, MPEG, etc. And sequentially stored in a predetermined data area.

(2) Feature amount detection step In this step, the feature amount detection unit 31 detects a sudden change or continuous state of an image or sound from a plurality of frame data stored in the frame memory 20, and is included in the moving image data. Understand the change characteristics of images and sounds.
Hereinafter, a feature amount detection method will be described.
(A) First Feature Quantity Detection Method As a first example of the feature quantity detection method, a technique for detecting the motion of a subject included in an image can be applied. This method divides each frame data image at different times into block areas, performs block matching processing on the reference image for each same block area, and moves the subject between frames from the coordinate position where the error is minimized. This applies a block matching method for detecting. Note that the block matching method is widely used in the IS11172-2 standard that has been internationally standardized by ISO / IECJTC1 / SC29 / WG11.

Here, the block matching method will be described.
FIG. 6 is a conceptual diagram showing a block matching method applied to the feature amount detection step.
For example, “The Institute of Television Engineers,“ Image Information Compression ”, Ohmsha, p. 92, 1991 ”describes a block matching method for detecting the motion of a subject included in moving image data composed of a plurality of continuous frame images. According to the above document, as shown in FIG. 6, an image of interest (hereinafter referred to as the current image) Fn and an image at the previous time (hereinafter referred to as the previous image for convenience) Fm. Pattern matching is performed for blocks (regions) Rn and Rm at specific positions. For example, the pattern matching method obtains the absolute value sum S of the difference between the pixel Xn in the block Rn and the pixel Xn-i in the block Rm shifted by i pixels according to the following equation {1}. The motion amount is detected by searching for the value sum S, that is, the shift position i that minimizes the evaluation amount.
S = Σ | Xn−Xn−i |... {1}
Here, the sum calculation Σ in the expression {1} is executed for all the pixels Xn belonging to the block Rn.
As described above, in the block matching method, the current image is divided into blocks, and a similar position to the previous image is searched by pattern matching for each block, so that the motion of the subject included in the series of images can be detected. .

(B) Second Feature Quantity Detection Method As a second example of the feature quantity detection method, a technique for calculating a change in pixel characteristics of a specific region included in an image can be applied. In this method, first, Laplacian processing is performed on the luminance component included in the image, the zero intersection point of the processed image is detected as a region boundary line, and a portion (subject region) closed by the continuous boundary line is defined as a specific region. Extract. Alternatively, a color component included in the image is analyzed, and a continuous portion with a small hue change is replaced with a representative color and extracted as a specific region. Then, by calculating the amount of change between the images for the extracted specific region, the amount of movement for the entire region can be obtained. In addition, by setting the specific area for the entire image and calculating the amount of change, it is possible to detect changes such as switching of shooting conditions (scenes).

(C) Third Feature Amount Detection Method As a third example of the feature amount detection method, a method for calculating a change in audio characteristics such as an audio level and a frequency band accompanying an image can be applied.
FIG. 7 is a schematic diagram showing a change in audio characteristics. Here, FIG. 7A is a schematic diagram showing a change in the level of audio accompanying each image, and FIG. 7B is a schematic diagram showing a change in the frequency band of audio.
As shown in FIG. 7A, in the case of detecting the level of sound accompanying the images F0 to F10 as the feature amount, the level L2 of the engine sound when the vehicle X is running, and the level of the brake sound when the vehicle X is stopped Changes in the sound characteristics of L4, level L6 at the time of horn sounding, level L8 of the engine sound at the time of vehicle Y departure, and level L10 of the engine sound at the time of vehicle X departure are represented by sound data A0, A2, A4, A6, A8, Detected based on A10.

On the other hand, as shown in FIG. 7B, in the case where the frequency band of the audio data A0 to A10 accompanying the images F0 to F10 is detected as the feature amount, the frequency B2 of the engine sound when the vehicle X is running, the vehicle X Changes in the sound characteristics of the stop sound frequency B4, the horn sounding frequency B6, the engine sound frequency B8 when the vehicle Y departs, and the engine sound frequency B10 when the vehicle X departs are the sound data A0, A2. , A4, A6, A8, A10.
Therefore, it is possible to detect the movement between vehicles by monitoring the level of sound between successive images, the change to a silent state, the transition of a frequency band, and the like.
Note that the above-described feature amount detection method shows an example applied to the present invention, and does not limit the embodiment of the present invention. Further, these feature detection methods may be used alone or in appropriate combination.

(3) Display Image Selection Step In this step, the display image selection unit 32 compares the detected feature amount with a preset threshold value or allowable range, and features that exceed the threshold value or allowable range. When the amount appears, an image of the frame data having the feature amount is selected as a display image or a processing target image in an image processing step described later.
FIG. 8 is a conceptual diagram showing an example of a display image selection step for selecting frame data based on image change characteristics.
As shown in FIG. 8, from the series of frame images F0 to F10 stored in the frame memory 20, the movement amount of the vehicles X and Y determined by the above-described feature amount detection step, and the presence or absence of the vehicles X and Y as subjects. And the like are compared with a preset threshold value, and a frame image with a change in photographing condition or a rapid movement of the subject is selected and extracted. For example, for a vehicle X, when a running state at a constant speed is detected in the frame image F2, and a sudden change in running speed due to sudden braking is detected in the frame image F4, the amount of change is set to a preset threshold value or If the amount of change is greater than the amount of change between the other frame images, the frame image F4 is determined to be an image having high topicality in the moving image data and is selected.

Further, paying attention to the existence of the vehicle X as a subject, when there are a frame image F0 in which the vehicle X does not exist in the frame and frame images F2 to F10 in which the vehicle X always exists, before and after the vehicle X appears. The frame images F0 and F2 are selected by determining that the images have high topicality in the moving image data.
Note that the frame data selection method based on the change characteristics of the image is not limited to the above-described method, and a method such as a rapid change in luminance or saturation of the subject may be used. In short, what is necessary is just to be able to monitor the feature quantity that can grasp the change of the photographing situation and the movement of the subject.

Next, an example of a display image selection step for selecting frame data based on the sound change characteristics will be described with reference to FIG.
As shown in FIG. 7, since the movement of the vehicles X and Y in the moving image data and the change in the shooting situation are also recorded as sound, the engine generated by the vehicles X and Y and the like found by the feature amount detection step described above. A sound level such as sound or horn, a frequency band, or the like is compared with a preset threshold value, and a frame image having a change in shooting state or high topicality is selected and extracted. For example, the sound level emitted from the vehicles X and Y is shown as shown in FIG. 7A. The sound level L6 at the time of horn sounding in the sound data A6 is the level L2 of the running sound of the vehicle X in the sound data A2. If it is extremely large compared to a preset threshold value (allowable range), the frame image F6 corresponding to the audio data A6 is determined to be an image having high topicality in the moving image data and selected. .

When attention is paid to the frequency bands emitted from the vehicles X and Y, as shown in FIG. 7B, the frequency B4 of the brake sound accompanying the sudden stop of the vehicle X in the audio data A4 is the vehicle X in the audio data A2. When the frequency is very high compared to the frequency B2 of the running sound and the threshold value (allowable range) set in advance, the frame image F4 corresponding to the audio data A4 is an image having high topicality in the moving image data. Judgment and selection.
Note that the frame data selection method based on the sound change characteristics is not limited to the above-described method, and a rapid change in the sound level or frequency or the duration of the same sound state (for example, a silent state). Etc. may be used.

(4) Image processing step In this step, the selected frame image is processed by the image processing unit 33 into a predetermined expression format that emphasizes the change in the shooting situation and the movement of the subject, and the monitor 50 and the printer 60 are used. Display output. Here, the image displayed and output on the monitor 50, the printer 60, or the like may change the enhancement processing of the image enhanced by the image processing unit 33 according to the degree of change in the feature amount. . Further, as a display method on the monitor 50, the printer 60, or the like, in addition to a method of displaying the emphasized images together with other images, display and output may be performed with priority over other images. Furthermore, at the time of image display output in scroll display, switching of shooting conditions or a sudden movement of a subject may be notified and recognized by flashing, character display, alarm, or the like.
Hereinafter, enhancement processing applied to the image processing step will be described.

(A) First Enhancement Processing FIG. 9 is a conceptual diagram showing an example of the first enhancement processing in the image processing step.
In the first enhancement processing, the shape of the display frame (frame) at the time of display output to the monitor 50, the printer 60, etc. is changed to a unique shape different from the normal for the frame image selected by the display image selection unit 32. It is something to be made.
As illustrated in FIG. 9, for example, when the display image selection unit 32 selects frame images F4, F6, and F10 having high topics such as sudden braking, horn sounding, and sudden starting in the movement of the vehicle X, By changing the shape of the display frame of the frame image to a normal square or rectangle, and applying a trapezoid, parallelogram, or a pre-prepared mount frame, etc., the display was emphasized for each change in the movement of the vehicle X A still image can be displayed and output.

(B) Second Enhancement Processing FIG. 10 is a conceptual diagram showing an example of the second enhancement processing in the image processing step.
In the second enhancement process, the display size at the time of display output is changed to a size different from the normal size for the frame image selected by the display image selection unit 32.
As shown in FIG. 10, for example, when the display image selection unit 32 selects a frame image F6 having high topicality such as honking in the movement of the vehicle X, it is displayed larger than the other frame images. By changing the display size, it is possible to display and output a still image that emphasizes the change in the movement of the vehicle X.

(C) Third Enhancement Processing FIG. 11 is a conceptual diagram showing an example of the third enhancement processing in the image processing step.
The third emphasis process is to change the display size and the rate of deformation of the display frame of the frame image selected by the display image selection unit 32 according to the degree of change of the feature amount.
As shown in FIG. 11, for example, the display size is changed stepwise or in an inclined manner corresponding to the level of the sound level emitted by the vehicle X detected by the feature amount detection unit 31. As an example, the audio level shown in FIG. 7 is calculated as a magnification with respect to the reference level, and the magnification (for example, 1.2 times) is applied to the reference display size as it is. As a result, the voice data A4 and A6 having a high voice level such as a sudden braking sound or horn sound generated by the vehicle X are processed into a display size corresponding to the voice level, and the change characteristics of the voice Can be displayed and output as a still image. Here, an example in which the sound level of the subject is applied as a feature amount for determining the degree of enhancement processing is shown, but this is only an example and does not limit the embodiment of the present invention. Therefore, the degree of enhancement processing may be changed according to the amount of movement of the subject.

Note that the image processing method is not limited to the above-described emphasis processing, but emphasizes the image quality and display gradation of the selected frame image as compared to the normal display or changes the display position. May be.
According to the present embodiment having the series of steps described above, the switching of the shooting situation in the moving image data and the movement of the subject are detected as a change in the feature amount included in the image or sound, and the moving image data is detected based on the feature amount. The representative image can be selected and highlighted, so that a still image can be easily extracted from the moving image data and displayed and output in an image representation that allows the captured content to be intuitively recognized. In this embodiment, when the feature amount detected in the feature amount detection step shows a predetermined change, a frame image having the feature amount is selected as a display image in the display image selection step, and (image processing is performed). However, the present invention is not limited to this. The present invention is not limited to this, and only the frame image showing the change in the predetermined feature value is processed in the feature value detection step, A frame image that has not been processed may be displayed and output sequentially or collectively.

<Second Embodiment>
Next, a second embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings.
In this embodiment, in the first embodiment described above, the user of the image processing apparatus arbitrarily selects and sets the feature amount and the image processing method.
That is, in the image processing apparatus shown in FIG. 1, the user of the image processing apparatus operates the input unit 70 or through a communication line (not shown) or the like, a feature amount selection command and a processing method selection command. Is used to select a frame image that satisfies a desired condition based on either a change in the feature quantity of the image signal, a change in the feature quantity of the audio signal, or a change in the feature quantity of the image signal and the audio signal. Further, the processing method of the frame image is selected, and the expression format on the monitor 50, the printer 60, etc. is arbitrarily set.

Hereinafter, the processing operation of this embodiment will be described with reference to the flowchart of FIG. The description of steps equivalent to those in the first embodiment described above will be simplified.
As shown in FIG. 12, in steps S201 and S202, a series of frame images and accompanying audio information are selected and extracted from moving image data at predetermined time intervals, subjected to compression encoding processing, and subjected to predetermined data in the frame memory 20. Store in the storage area. Next, in steps S203, S204, and S205, a change in the feature amount selected and instructed by the user via the input unit 70 or the like is detected for the series of frame images and audio stored in the frame memory 20, and the image or Understand the change characteristics of voice. That is, for captured and recorded moving image data, a frame image having a desired shooting content is selected and displayed in a later-described step on the basis of a change in a feature amount designated by a user, that is, a change characteristic of an image or sound. Is output.

  Next, in step S206, the display image selection unit 32 selects a frame image including switching of shooting conditions, a sudden movement of the subject, and the like based on the above-described image or sound change characteristics, and in steps S207 and S208. The selected frame image is subjected to image processing selected and instructed by the user via the input unit 70 or the like. Here, an arbitrary processing method is selected from among a plurality of image processing processes including the above-described enhancement processing. For example, a frame image indicating a change in shooting state or a change in a subject is displayed with a display size larger than that of a normal frame image. Control the display frame such as increasing the size, changing the display frame, moving the display position to the center of the monitor screen, emphasizing the brightness and saturation of the display screen, and controlling the display image quality such as flushing. .

In step S209, the frame image processed by the image processing unit 33 is displayed and output to the monitor 50, the printer 60, a facsimile apparatus, a PDA, a personal computer, or the like connected via a communication line. According to the present embodiment having the above series of steps, the user can select an arbitrary feature amount and image processing, so that a frame image having a desired shooting content (change in feature amount) can be visually recognized. It can be displayed and output in a highly expressive format. In the present embodiment, the case where the user arbitrarily selects and sets both the feature amount and the image processing method has been described. However, the target to be selected and set may be either the feature amount or the image processing method.
By the way, the image processing apparatus and the image processing method shown in each embodiment can be incorporated into an image processing apparatus such as a video player, a facsimile apparatus, and a printer in addition to the above-described video camera, electronic still camera, and personal computer, or application software. It goes without saying that it can be realized well by providing as.

1 is a block diagram showing a first embodiment of an image processing apparatus according to the present invention. It is a block diagram which shows schematic structure of a moving image data taking-in part. It is a conceptual diagram which shows the internal area | region of a frame memory. It is a flowchart which shows the processing operation in 1st Embodiment. It is a conceptual diagram which shows a moving image data taking-in step. It is a conceptual diagram which shows the block matching method applied to a feature-value detection step. It is a schematic diagram which shows the change of an audio | voice characteristic. It is a conceptual diagram which shows an example of the display image selection step which selects frame data based on the change characteristic of an image. It is a conceptual diagram which shows the example of the 1st emphasis process in an image process step. It is a conceptual diagram which shows the example of the 2nd emphasis process in an image process step. It is a conceptual diagram which shows the example of the 3rd emphasis process in an image process step. It is a flowchart which shows the processing operation in 2nd Embodiment. It is a conceptual diagram of the method of selecting and displaying arbitrary frame images from a plurality of frame images constituting moving image information.

Explanation of symbols

10 Movie data fetching unit 20 Frame memory 30 MPU
31 feature quantity detection unit (feature quantity detection means)
32 Display image selection unit (frame image selection means)
33 Image processing unit (frame image processing means)
40 hard disk 50 monitor (image output means)
60 Printer (image output means)
70 Input unit (feature quantity selection means, processing method selection means)
80 bus

Claims (8)

A feature amount detecting means for detecting a change in the feature amount included in a plurality of frame images constituting the moving image information;
Frame image selection means for selecting a specific frame image from the plurality of frame images based on the change in the feature quantity detected by the feature quantity detection means;
Frame image enhancement means for performing an enhancement process for changing a display frame shape on the specific frame image selected by the frame image selection means;
Image output means for displaying and outputting, as still images, the plurality of frame images including the specific frame image subjected to enhancement processing for changing a display frame shape by the frame image enhancement means ;
An image processing apparatus comprising: A feature amount detecting means for detecting a change in the feature amount included in a plurality of frame images constituting the moving image information;
Frame image selection means for selecting a specific frame image from the plurality of frame images based on the change in the feature quantity detected by the feature quantity detection means;
Frame image enhancement means for applying a predetermined enhancement process to the specific frame image selected by the frame image selection means;
Print output means for printing out the plurality of frame images including the specific frame image subjected to enhancement processing by the frame image enhancement means as still images;
An image processing apparatus comprising: Moving image information input means for inputting moving image information including a plurality of frame images;
Movie information extraction means for extracting frame images at predetermined time intervals from the movie information input by the movie information input means;
Feature amount detection means for detecting changes in feature amounts included in a plurality of frame images extracted by the moving image information extraction means;
Frame image selection means for selecting a specific frame image from the plurality of frame images based on the change in the feature quantity detected by the feature quantity detection means;
Frame image enhancement means for applying a predetermined enhancement process to the specific frame image selected by the frame image selection means;
Image output means for displaying and outputting, as still images, a plurality of frame images extracted by the moving image information extraction means, including the specific frame image subjected to enhancement processing by the frame image enhancement means;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the feature amount is a movement of a subject included in a plurality of frame images constituting the moving image information.   The image processing apparatus according to claim 1, wherein the feature amount is audio information associated with each of a plurality of frame images constituting the moving image information. Feature quantity detection means for detecting the degree of change in sound accompanying each frame image, which is a component of video information;
Frame image enhancement means for changing the form of each frame image, which is a component of the moving image information, according to the degree of change of the audio information detected by the feature amount detection means;
Image output means for displaying and outputting each frame image including a frame image whose form has been changed by the frame image enhancement means as a still image;
An image processing apparatus, wherein the plurality of frame images output by the image output means have different forms. Causing the feature amount detection unit to detect a change in the feature amount included in the plurality of frame images constituting the moving image information;
A step of causing a display image selection unit to select a specific frame image from the plurality of frame images based on a change in the feature amount detected by the feature amount detection unit ;
Causing the image processing unit to execute a process of performing an enhancement process for changing a display frame shape on the specific frame image selected by the display image selection unit;
A step of causing the image output unit to display and output the plurality of frame images including the specific frame image subjected to the enhancement processing by the image processing unit ;
An image processing method comprising: Causing the feature amount detection unit to detect a change in the feature amount included in the plurality of frame images constituting the moving image information;
A step of causing a display image selection unit to select a specific frame image from the plurality of frame images based on a change in the feature amount detected by the feature amount detection unit;
Causing the image processing unit to execute a process of performing a predetermined enhancement process on the specific frame image selected by the display image selection unit;
Printing out the plurality of frame images including the specific frame image subjected to the enhancement processing by the image processing unit as a still image;
An image processing method comprising:

Images