JPH10164503A - Video-processing device and method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain nonuniform easy-to-see dynamic images, without frame omission and to obtain the easy-to-see dynamic images at a high frame rate by resetting the frame rate of still images corresponding to the fluctuation of the throughput of a video processing means at a certain time interval matched with the average of the throughput. SOLUTION: A video-processing means 103 inputs source video information outputted by a source video output means 102, performs a compression encoding processing, outputs it as encoded video information and is operated on a multi- task operating system. A maximum video-processing time detection means 104 measures the time when the video-processing means 103 processes the still images and obtains the maximum processing time in a section, until measurement reset signals 109 are inputted in it. Thus, the frame rate by which the video processing means 103 can process all the still images without omitting frames at least at the time of the frame rate is outputted as video throughput information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a video processing apparatus for inputting and processing moving image information.

[0002]

2. Description of the Related Art There are various configurations of a video processing device for inputting and processing moving image information. A typical example is a moving image input as analog video from a video camera or the like. A so-called video capture device or the like, which converts this into an information compression process and stores it, is conceivable. In such a video capture device, since a moving image is input in real time, the information compression processing is usually performed in real time. That is, when a moving image composed of 15 still images per second is input, the information compression processing means is required to surely compress the 15 still images per second. For this reason, in such a video capture device, the information compression processing means is usually realized using dedicated hardware.

An example of the configuration of a video processing device (video capture device) according to the prior art will be described with reference to FIG. In the figure, reference numeral 2101 denotes a video camera, which captures video information,
It is output as analog video information. Reference numeral 2102 denotes an original video output unit, which receives analog video information output from the video camera (2101), is composed of discrete digital data, and is a digital video composed of a plurality of still image information representing still images per unit time. It is output as original video information as information. Original video output means in this embodiment
(2102) outputs original video information consisting of 30 fps, that is, 30 still images per second. That is, the "unit time" is about 0.033 seconds in this embodiment. Two
Reference numeral 103 denotes a video processing unit that receives the original video information output from the original video output unit (2102), performs a compression encoding process, and outputs the encoded video information. In this configuration example, it is assumed that the video processing means (2103) is realized using hardware, and can process real video information output from the raw video output means (2102) in real time. That is, the image processing means (2
103) has the ability to complete the compression encoding process of one still image in 0.033 seconds or less. FIG. 22 is a diagram showing a flow of information in a video processing device according to a conventional technique. The details of the reference numerals in the figure are omitted because they have already been described above. FIG. 23 is a flowchart illustrating an example of processing of the video processing device according to the related art.

[0004] The operation of the conventional image processing apparatus configured as described above will be described below.

1 In step 1 of FIG. 23, if there is no end instruction from the user, step 2 is executed.

2. In step 2, the video camera (2101) captures video information with audio and outputs it as analog video information. The original video output means (2102) receives the output analog video information, performs digital conversion, and outputs it as one piece of still image information constituting the original video information.

3 In step 3, the image processing means (2103)
Receives the still image information output in step 2, performs compression encoding in 0.033 seconds, and outputs it as encoded video information. Returning to step 1, if there is no end instruction, step 2 is executed.

[0008] In step 2, after 0.033 seconds from 2 again, the original video output means (2102) again outputs still image information in the same manner as 2.

5 In step 3, video processing means (2103)
Processes the still image information output at 4 in the same manner as 3
Output as encoded video information.

6 The above is repeated as appropriate (end). In addition, in order to clarify the problems with the conventional technology,
The operation of the second configuration example according to the related art will be described.
The second configuration example has exactly the same configuration as the above-described first configuration example according to the related art, except that the video processing means (2103)
However, the only difference is that they operate as one process that operates on a multitask operation system. The video processing means (2103) in this configuration example is assumed to have a processing capability of 30 fps alone as in the first configuration example, but shares computer resources with other processes on the multitask operation system. Therefore, if it is operated at the same time as another process, it is assumed that the processing capacity is significantly reduced. 24 is a diagram 2401 in FIG. 24 showing still image information constituting the original video information processed by this embodiment. Also, the figure showing the playback frame interval of the finally obtained encoded video information is
2402.

The operation of the second image processing apparatus according to the prior art configured as described above will be described below with reference to FIGS.

1 In step 1 of FIG. 23, if there is no end instruction from the user, step 2 is executed.

2. In step 2, the video camera (2101) captures video information with audio and outputs it as analog video information. The original video output means (2102) receives the analog video information output in step 1, performs digital conversion, and outputs it as one piece of still image information constituting the original video information
(A of the original video information (2401)).

3 In step 3, the image processing means (2103)
Receives the still image information output in step 2, performs compression encoding in 0.033 seconds, and outputs it as encoded video information. Returning to step 1, if there is no end instruction, step 2 is executed.

4 Here, another process running on the operating system enters an operating state and robbs a part of computer resources.

5 In step 2, after 0.033 seconds from 2 again, the original video output means (2102) again outputs still image information (b in the original video information (2401)), similarly to 2.

6 In step 3, the image processing means (2103)
Processes the still image information output in step 5, the processing of this still image takes 0.1 seconds due to the influence of other processes.

7 Return to step 2, but at this point
It takes 0.1 seconds. Therefore, from 5
Video information input within 0.1 seconds is discarded.
That is, here, two still images are discarded (c and d of the original video information (2401)). Then, the still image information three frames ahead, which is equivalent to the time after 5 to 0.1 seconds, is output (the original video information
1) e).

8 Here, another process running on the operating system enters a hibernation state to release computer resources.

9 In step 3, the image processing means (2103)
Processes the still image output in 7 in 0.033 seconds and outputs it as encoded video information.

10. Here, another process operating on the operating system again enters the operating state, and takes away a part of the computer resources.

11 In step 2, the original video output means (210
2) outputs still image information (the original video information (2401)
f). However, in step 3, the video processing means (2103) is affected by another process,
Takes seconds. Therefore, the subsequent still image is discarded (g of the original video information (2401)).

12 In step 2, the original video output means (210
2) outputs still image information (the original video information (2401)
h). However, the video processing means (2103) is affected by other processes, and it takes 0.1 seconds to process this still image. Therefore, the still image that follows is discarded (i, j in the original video information (2401)).

13 The above is repeated as appropriate (end). Further, a method of controlling a frame rate according to a load of video processing in a “multimedia device” disclosed in Japanese Patent Application Laid-Open No. 5-64186 (hereinafter, Prior Art 1) will be described. FIG. 25 is a configuration diagram of a video processing apparatus according to the technology disclosed in Prior Art 1 (summarizing the essential parts of the embodiment actually disclosed in Prior Art 1). In the figure, reference numeral 2501 denotes an information reading unit which reads multimedia information (such as a digital moving image) from an optical disk or the like and sends it to an information buffering means (2502). Here, the multimedia information is treated as a digital moving image for simplicity, and the digital moving image is referred to as original video information. Reference numeral 2502 denotes information buffering means for temporarily storing the original video information sent from the information reading unit (2501). Reference numeral 2503 denotes an information conversion unit.If the video processing unit (2504) can process a video, the video buffering unit (2504)
The original video information is read from 2), deleted from the buffer, and output to the video processing means (2504) .If the load detection means (2505) outputs an overload signal, the video buffering means After changing the frame rate of one unit of the original video information read from (2502), it is output to the video processing means (2504). Reference numeral 2504 denotes an image processing means for processing an input image. Reference numeral 2505 denotes a load detecting means for outputting an overload signal when the amount of original video information stored in the information buffering means (2502) exceeds a certain value. FIG.
FIG. 3 is a diagram showing a flow of information in the video processing device according to the technique. The details of the reference numerals in the figure are omitted because they have already been described above. FIG. 27 is a flowchart illustrating an example of processing of the video processing device according to the above technique.

The operation of the second image processing apparatus according to the prior art configured as described above will be described below with reference to FIGS. 25 to 27.

1 In step 1 of FIG. 27, if there is no end instruction from the user, steps 2 and 3 are executed. Step 2 and subsequent steps and step 3 and subsequent steps are executed in a multitasking manner.

2 In step 2, the information reading means (250
1) reads the original video information from an optical disk or the like and temporarily stores it in the video buffering means (2502). Step 2
Thereafter, this is repeated until an end instruction is issued in step 1.

3 In step 3, the image processing means (2504)
Check if is now able to process the video. If it can be processed, the process proceeds to step 4.

4 In step 4, it is checked whether or not the original video information has been stored in the video buffering means (2502).

5 In step 5, information conversion means (2503)
Reads one unit of original video information from the video buffering means, deletes it from the buffer, and proceeds to step 6.

6 In step 6, load detection means (2505)
Checks the amount stored in the video buffering means (2502) and checks if it exceeds 80%. 80%
If not, go to step 8. In step 8, one unit of the read original video information is output to the video processing means (2504). Thereafter, the process returns to step 3 and is repeated.

6 If the original video information is stored in the video buffering means (2502) at 80% or more in the step 6, the load detecting means (2505) outputs an overload signal, and proceeds to the step 7. In step 7, the information converting means (2503) which has received the overload signal reads one unit of the original video information from the video buffering means, deletes this from the buffer, and converts The frame rate is converted so as to become the rate, and the process proceeds to step S8. In step 8, one unit of the original video information is output to the video processing means (2504). Thereafter, the process returns to step 3 and repeats (end).

[0033]

The video processing apparatus according to the prior art simply processes the output original video information one by one as described in the first configuration example. Therefore, in a configuration in which the original video information is output at 30 fps, the video processing means also needs to have the ability to process the original video information at 30 fps. The configuration example in which the video processing means is realized by one process on the multitask operation system is the above-described second configuration example. In this configuration example, the original video information is output at 30 fps. In other words, the image processing means can process only 15 fps on average because other processes take away computer resources. That is, even if ten pieces of still image information constituting the original video information are output, only five of them can be processed. In this configuration example, when the encoded video information is created by setting the code of the discarded still image to be “same as the previous still image”, the encoded video information and the encoded video information as shown by 2402 in FIG. Become. This encoded video information (2402)
Is composed of a total of five still images, but the playback time of each still image varies, so playing such coded video information makes it hard to see It becomes a moving image.

As described above, when an image processing apparatus according to the prior art is to be realized by a process on a multitask operation system, computer resources are deprived by other processes, and when image processing of original image information cannot be made in time, Uneven video breaks occur, and when the processed video information is played back later, the parts where the moving images move smoothly and the parts that do not move appear alternately, making the moving images jerky and difficult to see. Occurs. This problem is also caused by the problem that when the computer resources are deprived of other processes and the image processing capacity is chronically insufficient, an unbalanced image break occurs as in the second configuration example described above. However, there may be another problem that another process suddenly operates and a large video break occurs temporarily. Also, even if no other process exists,
If the processing capacity of the video processing means is marginal compared to the amount of input original video information, and if the quality of the input original video information is unsuitable for processing, May not be in time, the still image may be discarded, and an uneven frame may be dropped, resulting in an unsightly moving image. For example, when the video processing means performs video compression processing, when original video information that is difficult to compress is input, it takes time to process the original video information, and uneven frame dropping occurs. There is.

Further, although the processing capability of the video processing means is sufficient, another problem may occur that the video processing means does not process a part of the still image, resulting in uneven dropout of frames. sell. For example, when performing video processing such as compression encoding of a moving image, in order to maintain a predetermined encoding bit rate, a still image that cannot be compressed in bit rate (the encoding of the still image May become too large and the video processing means may discard the still image, for example, in which the determined bit rate cannot be maintained. In this case, if the video processing unit discards the still image at a timing dependent on the function of maintaining the bit rate, the discarding interval is not uniform, and the above-described jerky unsightly moving image is output. I will.

Further, as a problem common to all the above-mentioned problems, there is a problem that the capability of the apparatus is wasted by discarding the still image. As the original video output means, when capturing a video and outputting it, it is not known which still image is to be discarded, so for the time being, prepare all the still images for output (analog to digital conversion Or copy to a buffer in main memory). However, due to insufficient video processing capability or due to video processing (eg, maintaining a bit rate), some of the prepared still images are discarded without being used. In other words, the computer resources used to prepare the still images to be discarded are completely wasted,
This means that the operation is extremely inefficient from the viewpoint of the entire apparatus.

As a means for solving such a problem, as described above, the prior art 1 discloses a technique of measuring the load of video processing and adjusting the frame rate accordingly. However, this technique has insufficient points as described below in solving the above-mentioned problem.

The gist of this technique is that first, all the original image information (in the prior art 1, "multimedia information") is accumulated in a buffer on a FIFO, and the accumulated amount of the buffer is checked. , Detect the load of video processing,
When reading information from a buffer, the frame rate of the read information is changed depending on the load. This configuration has the following two problems in solving the above problem.

First, when the video processing capability is low, all the original video information that is finally discarded must be stored in the buffer. This inefficiently consumes enormous computational power, such as reading of original video information and copying to a buffer.

Second, in order to detect the capability of video processing,
Although the frame rate is controlled when the amount of the original video information exceeds a certain value by using the amount of the original video information stored in the buffer, this method cannot detect an accurate load of the video processing capability.

That is, if the amount of original video information input to the buffer and the amount that can be processed are very small, and the amount that can be processed is smaller, the amount of original video information stored in the buffer is It will increase slowly. In this case, the frame rate may be reduced slightly. On the other hand, if the amount of input original image information and the amount of image processing that can be processed are extremely small, the amount of original image information stored in the buffer increases at a high speed. Will go. In this case, the frame rate must be greatly reduced. However, if control is performed only based on whether or not the amount of original video information stored in the buffer exceeds a certain value, the same control is performed for the above two patterns, which is a very inconvenient situation. In other words, although the difference between the input amount and the processing amount of the original video information is slight, the accumulated amount exceeds a certain amount, and the input amount suddenly becomes extremely small. Although the input amount and the processing amount of the video information are overwhelmingly large, the input amount is reduced only a little, and the buffer overflows without control. Also, in order to detect long-term fluctuations in video processing capability, the buffer must be large. If control is to be performed using a small buffer, the frame rate is set to a low value as soon as the accumulated amount exceeds a certain amount for at least one image, and the frame rate is increased to a higher value if even one image falls below a certain amount. Since the setting is performed, a jerky and difficult-to-see moving image is eventually output.

According to the present invention, in view of the above-mentioned problems of the conventional video processing apparatus, the video processing means is realized as a process that operates on a multitasking operating system. The purpose of the present invention is to prevent image breaks from becoming uneven even when they occur in the processing means, and to prevent large image breaks from occurring even when a sudden lack of computational power occurs. It is another object of the present invention to prevent the occurrence of uneven frame drop due to the image processing, and to perform the control for solving the above problem even if the buffer amount is small or the buffer does not exist. Is what you do.

[0043]

According to a first aspect of the present invention, there is provided a video processing apparatus comprising: a plurality of pieces of still image information representing a still image at a still image time interval which is an arbitrary time interval; When the original video information is output, a video processing unit that processes the original video information, and a frame rate control time interval that is an arbitrary time interval according to the degree of progress of the processing of the video processing unit. Frame rate information output means for obtaining and outputting original video frame rate information representing the number of still images, and outputting the original video information and outputting the original video frame rate information and outputting the original video frame rate information in accordance with the original video frame rate information. Original video output means for changing the still image time interval of the original video information.

Further, the video processing apparatus according to claim 2 is
2. The video processing device according to claim 1, wherein the video processing unit obtains a processing time for processing one piece of still image information, and obtains a maximum value of the processing time until a measurement reset signal is transmitted. Maximum video processing time detecting means for obtaining and outputting video processing capability information and frame rate information output means transmit the measurement reset signal at arbitrary time intervals, and the maximum video processing time at each frame rate control time interval. This is for obtaining and outputting original video frame rate information in accordance with the video processing capability information output by the detection means.

The video processing apparatus according to claim 3 is
2. The image processing apparatus according to claim 1, wherein an average processing amount of the original image information per unit time of the image processing means in a section until the measurement reset signal is transmitted is obtained, and image processing capability information is obtained and output from the average processing amount. The average video processing capability detection means and the frame rate information output means transmit the measurement reset signal at arbitrary time intervals, and the video processing output by the average video processing capability detection means at every frame rate control time interval It obtains and outputs the original video frame rate information according to the capability information.

The video processing device according to claim 4 is
4. The video processing apparatus according to claim 1, wherein when the video processing means cannot process the original video information, the video processing means can temporarily store the original video information and the video processing means can process the original video information. In the case of (1), a video buffering means for outputting the original video information accumulated most recently to the video processing means is provided.

The video processing device according to claim 5 is
5. The video processing apparatus according to claim 4, wherein the maximum video processing time detecting means or the average video processing capacity detecting means obtains and outputs the video processing capacity information in accordance with a change in the amount of the original video information stored in the video buffering means. It is.

The video processing apparatus according to claim 6 is
5. The video processing apparatus according to claim 4, wherein the frame rate information output means, when obtaining the original video frame rate information, depends not only on the video processing capability information but also on the storage amount of the original video information stored in the video buffering means. It obtains and outputs original video frame rate information.

The video processing device according to claim 7 is
The video processing apparatus according to claim 4, further comprising a frame rate control time interval setting means for changing the frame rate control time interval according to the free space of the buffer of the video buffering means.

The video processing apparatus according to claim 8 is
The video processing device according to claim 1,
Initial value setting means for judging the performance of the apparatus itself and obtaining and outputting calculation capability information according to the result of the judgment; and frame rate information output means, wherein the initial value setting means is provided before the video processing means starts video processing. And calculates and outputs the original video frame rate information in accordance with the computational function information output by.

The video processing method according to claim 9 is
When the original video information including a plurality of still image information representing a still image for each still image time interval that is an arbitrary time interval is output, a video processing step of processing the original video information,
A frame rate information output step of obtaining and outputting original video frame rate information representing the number of still images per unit time for each frame rate control time interval that is an arbitrary time interval according to the degree of progress of the processing of the video processing step; Outputting the original video information and changing the still image time interval of the original video information according to the original video frame rate information when the original video frame rate information is output. .

According to a tenth aspect of the present invention, in the video processing method according to the ninth aspect, the video processing step determines a processing time for processing one piece of still image information and resets a measurement of the processing time. A maximum video processing time detection step of obtaining and outputting video processing capability information from the maximum value until a signal is transmitted and a frame rate information output step transmits the measurement reset signal at an arbitrary time interval. In addition, at each frame rate control time interval, original video frame rate information is obtained and output according to the video processing capability information output by the maximum video processing time detecting step.

In the video processing method according to the present invention, the average processing of the original video information per unit time of the video processing step in the section until the measurement reset signal is transmitted is provided. An average video processing capability detection step of obtaining and outputting video processing capability information from the average processing volume, and a frame rate information output step: transmitting the measurement reset signal at an arbitrary time interval; The original video frame rate information is obtained and output in accordance with the video processing capability information output by the average video processing capability detection step at each interval.

According to a twelfth aspect of the present invention, in the video processing method of the ninth, tenth or eleventh aspect, when the video processing step cannot process the original video information, the original video information is temporarily stored. And when the video processing step can process the original video information, a video buffering step of outputting the oldest stored original video information to the video processing step is provided.

A video processing method according to a thirteenth aspect of the present invention is the video processing method according to the twelfth aspect, wherein the maximum video processing time detecting step or the average video processing capability detecting step includes the step of: The video processing capability information is obtained and output according to a change in the amount of information.

According to a fourteenth aspect of the present invention, in the video processing method according to the twelfth aspect, the frame rate information output step includes, when obtaining the original video frame rate information, not only the video processing capability information but also the video buffer information. The original video frame rate information is obtained and output according to the storage amount of the original video information stored in the ring step.

According to a fifteenth aspect of the present invention, in the video processing method according to the twelfth aspect, the frame rate control time interval setting step changes the frame rate control time interval in accordance with the free space of the buffer in the video buffering step. It is provided with.

According to a sixteenth aspect of the present invention, in the video processing method according to any one of the ninth to fifteenth aspects, the performance of the apparatus itself is determined, and calculation capability information is obtained and output according to the result of the determination. The initial value setting step and the frame rate information output step obtain and output the original video frame rate information in accordance with the calculation capability information output by the initial value setting step before the video processing step starts the video processing.

[0059]

BEST MODE FOR CARRYING OUT THE INVENTION

(Embodiment 1) The video processing apparatus according to Embodiment 1 of the present invention obtains the maximum value of the time spent for video processing performed by the video processing means, and determines the frame rate using the maximum value as the interval between still images. When the output frame rate of the video information is set, the average frame rate is reduced, but coded video information that is easy to see without uneven frame drop is obtained.

FIG. 1 is a configuration diagram of a video processing device according to the first embodiment of the present invention. In the figure, reference numeral 101 denotes a video camera which captures video information and outputs it as analog video information. Reference numeral 102 denotes an original video output unit, which receives analog video information output from the video camera (101), is composed of discrete digital data, and is a digital video composed of a plurality of pieces of still image information representing still images per unit time. It is output as original video information as information. When the original video frame rate information (107) is input, the "unit time" is changed according to the original video frame rate information (107). The original image output means (102) in the present embodiment outputs original image information consisting of 30 still images per second at 30 fps, that is, 30 frames per second. That is, the “unit time” is about 0.033 seconds before the input of the original video frame rate information (107). Reference numeral 103 denotes a video processing unit, which inputs the original video information output by the original video output unit (102), performs compression encoding processing, and outputs the encoded video information, and operates on a multitasking operating system. Is what you do.
104 is a maximum video processing time detecting means,
(103) measures the time for processing the still image, and in the section until the measurement reset signal (109) is input,
The maximum processing time is determined, and the video processing means (10
If 3) is at least the frame rate, a frame rate at which all still images can be processed without dropping frames is output as video processing capability information (105). Therefore, the video processing capacity information (105)
Is 10 fps, the interval between still images is 0.1 second, so that at least 0.1 seconds indicates that the video processing means (103) can process one still image.

That is, the worst value of the processing time so far is 0.1 second. This worst value is reset by inputting a measurement reset signal, and after resetting, the worst processing time is calculated until the next measurement reset signal is input. Reference numeral 106 denotes frame rate information output means, and for each frame control time interval (108), video processing capability information (105) output by the maximum video processing time detection means (104).
And obtains and outputs original video frame rate information (107). In this configuration example, the video processing capability information (105)
Is output as original video frame rate information (107) as it is. That is, the video processing capability information at that time (105)
Was 10 fps, the original video frame rate information (107)
Is also 10fps. Further, a measurement reset signal (109) is output to the maximum video processing time detecting means at each frame control time interval. In this embodiment, it is assumed that the frame control time interval (108) is 0.33 seconds. This value is set in advance. FIG. 2 is a diagram showing a flow of information in the video processing device according to the first embodiment. In the figure, 105 is video processing capability information, 107 is original video frame rate information, 108 is a frame control time interval, and 109 is a measurement reset signal. The details of each are omitted since they have already been made in the description of FIG. FIG. 3 is a flowchart illustrating an example of processing in the video processing device according to the first embodiment. FIG. 4 is a diagram showing still image information constituting the original video information processed by this embodiment.
01. Reference numeral 402 denotes a reproduction frame interval of the finally obtained encoded video information.

The operation of the video processing apparatus configured as described above will be described below with reference to FIGS.

1 In step 1 of FIG. 3, if there is no end instruction from the user, step 2 is executed. Step 2
Then, take in the video information from the video camera (101),
The original video output means (102) converts it into digital information,
It is output as one piece of still image information constituting the original video information (a of the original video information (401)).

(2) In step 3, the image processing means (103)
Receives the still image information output in step 1, performs compression encoding in 0.033 seconds, and outputs it as encoded video information. In step 4, the maximum video processing time detecting means (104)
Records the above processing time 0.033 seconds.

3. Here, another process operating on the operating system enters the operating state, and takes away a part of the computer resources.

4 In step 5, it is checked whether the current elapsed time has reached the frame rate control time interval (108). Since the default value of 0.33 seconds has not yet been reached, the process returns to step 1 and step 2.

5. In step 2, after 0.033 seconds from 1 again, the original video output means (102) outputs still image information again as in 1 (b in the original video information (401)). In step 3, the video processing means (103) processes this still image information, but it takes 0.1 seconds to process this still image due to the influence of other processes. In step 4, the maximum video processing time detecting means (104) sets the current video processing time to 0.03 seconds from the previous processing time of 0.033 seconds.
Since 1 second is longer, 0.1 seconds is recorded as the maximum processing time. Hereinafter, every time the video processing means (103) performs processing, the maximum video processing time detection means (104) records the maximum processing time. As in 4, step 5 to step 1,
Return to step 2.

6 In step 2, similarly to 5, the original video output means (102) outputs still image information. However, at this time, since it took a time of 0.1 seconds at 5, two still images output every 0.033 seconds were discarded without being output (c of the original video information (401)). , d). So 5 to 3
The still image corresponding to the still image of the next frame is output (e in the original video information (401)).

7. Here, another process operating on the operating system enters a hibernation state to release computer resources.

8 In step 3, the video processing means (103)
Processes the still image output in 6 in 0.033 seconds and outputs it as encoded video information. As in step 5, the process returns from step 4 to step 2.

9. Here, another process operating on the operating system enters the operating state again, and takes away a part of the computer resources.

In step 2, the original video output means (10
2) outputs still image information (f) of the original video information (401).
However, in step 3, the video processing means (103) is affected by another process, and it takes 0.0667 seconds to process this still image.

11 Returning from step 4 to step 2, at this point, since a time of 10 to 0.0667 seconds has elapsed, one still image has been discarded (the original video information (40
1) g). In step 2, the original video output means (102)
Is output (h in the original video information (401)).
In step 3, the video processing means (103) is affected by other processes and takes 0.1 seconds to process this still image. In step 4, this time is measured.

12 In step 5, the time here is 0.33
Seconds, the frame rate control time interval (10
Assuming that 8) has been reached, step 6 is executed. The maximum video processing time detecting means (104) is the processing time of the maximum still image in the processing up to this point is 0.1 second, and the frame rate of the still image interval of 0.1 second is 10 fps. 10 fps is output as information (105). The frame rate information output means (106) receives the video processing capability information (105) and outputs 10 fps as original video frame rate information (107). Upon receiving this, the original video output means sets the frame rate of the subsequent original video information to 10 fps. Further, the frame rate information output means (106) outputs a measurement reset signal (109) to the maximum video processing time detection means (104).

13 Returning to step 2, the original video output means
(102) outputs still image information. In step 3, the video processing means (103) is affected by other processes and takes 0.066 seconds to process this still image. In step 4, since the measurement reset signal (109) has been transmitted, this time is measured as the current processing time maximum value. Step 5 does not execute step 6 until the next time comes, and the process returns to step 2.

14 In step 2, the original video output means (10
In 2), the still image information is output every 0.1 seconds.
Sleep for 0.033 seconds. Thereafter, the original video output means (102) outputs the still image information. In step 3, the video processing means (103) is affected by other processes and takes 0.1 seconds to process this still image.

15 13 to 14 are appropriately repeated until the end instruction is confirmed in step 1 (end).

As described above, in the video processing apparatus according to the present embodiment, the maximum video processing time detecting means measures the processing time of the video processing means for processing a still image, Set the frame rate so that dropping does not occur. FIG. 4 is a diagram for explaining the coded video information obtained by this embodiment. The A section of the encoded video information (402) is 0.33 seconds in total, and there are five still images, so the average is 15 fps, but the encoding is performed in the same manner as in the configuration example using the conventional technology. The reproduction time of the still images constituting the video information varies and is a section in which the video is a jerky and hard to see video. In section B, the maximum processing time required for still image processing in section A is detected, and the processing time is reset to a frame rate having a still image interval. In this section B, since still images are output at 0.1 second intervals, the frame rate is 10 fps, which is lower than that of section A, but still images are always present at equal intervals. Therefore, the video is smoother and easier to see than section A.

As described above, in the video processing apparatus according to the present embodiment, at a certain time interval, in accordance with the fluctuation of the processing power of the video processing means, the video processing means is adjusted to the portion where the processing power is reduced most
Reset the frame rate of the still image. For this reason, when the processing capability of the video processing unit fluctuates, the interval between the still images of the encoded video information finally obtained fluctuates due to the fluctuation, and an unsightly video is output. The problem can be solved. That is, in the multitasking operating system, even if the processing capability of the video processing means fluctuates due to the influence of other processes, an easy-to-view video with uniform frame intervals can be obtained. Similarly,
When the processing capability of the video processing means is not sufficient and an original video that is difficult to process is input, the frame rate is reset based on the processing time required for processing the original video that is most difficult to process. As a result, an image with a uniform frame interval can always be obtained.

In this embodiment, the video processing capability information is directly used as the original video frame rate information.
The frame rate information output means may calculate and obtain the original video frame rate information based on the video processing capability information. For example, the original video output means is operating at 30 fps output of the original video information, but the processing capacity of the video is insufficient, so that the video is processed at 15 fps and the remaining original video information is discarded. If the original video output means is originally scheduled to output 15 fps of original video information and the original video output means operates, compared to the normal case, there is no waste, the entire system operates efficiently, and as a result, computer resources can be saved. In some cases, it is considered that the processing capability of the video processing means is also improved. Therefore, a frame rate larger than the value of the video processing capability information may be output as the original video frame rate information in anticipation of the improvement. For example, if the original video information is output at 30 fps and the ability to process the video is 15 fps, the frame rate of the original video information is not set to 15 fps, but the whole system is optimized It is possible to set at 20 fps, taking into account the performance improvement of.

In the present embodiment, since all still images cannot be processed on time at the originally set frame rate, an example in which the frame rate is reset to a lower value has been described. However, in this configuration example, it is obvious that the operation of returning the frame rate can be performed when other processes on the multitasking operating system are terminated and the video processing capability is improved. In other words, if the video processing ability improves,
In some cases, the processing time may take longer than 0.033 seconds. In such a case, the maximum video processing time detecting means may return the frame rate to 30 fps.

In the present embodiment, the measurement reset signal is transmitted at every frame rate control time interval. However, it can be transmitted at other intervals. For example, if it is desired to measure the maximum processing time at a longer interval, the measurement reset signal may be transmitted at a longer time interval.

Further, in the flowchart of FIG. 3, the algorithm of original video output → video processing → processing time measurement → frame rate change is adopted, but this is only an example, and the timing of the original video output is performed by interruption. Alternatively, the timing of changing the frame rate may be performed by a timer interrupt instead of sequentially, so that the operation does not necessarily need to be performed according to the flow of the flowchart in FIG.

In this embodiment, the output original video information is directly input to the video processing means. Of course, as described above, this configuration has a sufficient effect, but furthermore, a buffer such as a FIFO is provided between the original video output unit and the video processing unit, and in this embodiment, the discarded still image The information may be stored in a buffer and buffered. Then, as shown in the A section of 402 in FIG. 4 of the video,
A smooth section such as section B can be obtained without obtaining uneven sections. Also, it is possible to cope with a case where the video processing capability suddenly decreases.

Further, in the present embodiment, the maximum video processing time detecting means outputs the video processing capability information in the form of a frame rate. However, this does not matter even in other forms (for example, the maximum processing time itself). It goes without saying.

(Embodiment 2) The video processing apparatus according to Embodiment 2 of the present invention obtains the average of the video processing capability performed by the video processing means, and resets the output frame rate of the original video suitable for it. This is to obtain smooth encoded video information without uneven frame dropout.

FIG. 5 is a configuration diagram of a video processing device according to the second embodiment of the present invention. In the figure, reference numeral 501 denotes a video buffering means. When the video processing means (103) cannot process the original video information, the original video information is temporarily stored. When processing is possible, the original video information accumulated most recently is stored in video processing means (103).
Is output to Usually, it can be realized as a FIFO memory such as a ring buffer method. Reference numeral 502 denotes an average video processing capability detection unit, which measures the processing time for the video processing unit (103) to process a still image, and averages and processes a frame that can be processed within a section until a measurement reset signal is input. A rate is obtained and output as video processing capability information (503). In the present embodiment, the video processing capability information (5
03) represents the number of still image information pieces processed per unit time by the video processing means (103) by a frame rate. Therefore, if the video processing capability information (503) is 15 fps, it means that the video processing means (103) is processing 15 pieces of still image information per second on average. The average value is reset by inputting the measurement reset signal (109), and after resetting, the average processing capability in a section until the next measurement reset signal (109) is input is obtained. 504 is Embodiment 2
Means for outputting frame rate information, and means for detecting average video processing capability (50) at each frame control time interval (108).
The original video frame rate information (107) is obtained and output according to the video processing capability information (503) output by 2). In this configuration example, the video processing capability information (503) is output as it is as the original video frame rate information (107). That is,
If the video processing capability information (503) is 15 fps, the original video frame rate information (107) is also 15 fps. In this embodiment, the frame control time interval (108) is assumed to be 0.33 seconds. Also, for each frame control time interval (108),
A measurement reset signal (109) is output to the maximum video processing time detecting means. FIG. 6 is a diagram showing a flow of information in the video processing device according to the second embodiment. In the figure, 503
Is image processing capability information. The details of this code are omitted since they have already been made in the description of FIG.
Other reference numerals are the same as those in FIG. 1, and the description is omitted. FIG. 7 is a flowchart illustrating an example of processing in the video processing device according to the first embodiment. FIG. 08 is a graph showing the amount of original video information (the number of still image information) stored in the video buffering means. FIG. 9 shows a coded video information report finally obtained by this embodiment.

The operation of the video processing apparatus configured as described above will be described below with reference to FIGS.

1 If there is no end instruction from the user in step 1 of FIG. 7, steps 2 and 3 are simultaneously executed by multitasking. In step 2, the video camera (10
Captures video information from (1) and outputs it to the original video output means (1
02) is converted into digital information, and is set as one piece of still image information constituting original video information. At this point, the video processing means (10
3) can be processed, so the video buffering means (50
The information is output to the video processing means (103) without being stored in (1). Step 2 returns to step 1 and reaches step 2 again. In step 3, the video processing means (103) waits until the input of the original video information is performed, and executes step 4 when the input of the original video information is performed. In this case, since the original video information is immediately passed to the video processing means (103), the video buffering means does not accumulate anything (section A in FIG. 08). Then, the process proceeds to Step 4.

(2) In step 4, the image processing means (103)
Receives the still image information output in step 1, performs compression encoding in 0.033 seconds, and outputs it as encoded video information. In step 5, the average video processing capability detecting means (502)
Records the processing time 0.033 seconds, and counts 1 as the number of still images that could be processed.

3 In step 6, it is confirmed whether the current elapsed time has reached the frame rate control time interval (108). Since the default value of 0.33 seconds has not yet been reached, the process returns to step 1 and step 3.

4 Here, another process operating on the operating system enters the operating state, and takes away a part of the computer resources.

5 In step 2, after 0.033 seconds from 1 again, the original video output means (102) outputs still image information in the same manner as 1 again. In step 4 after step 3, the video processing means (103) processes this still image information, but it takes 0.1 seconds to process this still image due to the influence of other processes. In step 5, the average video processing capability detection means
(104) measures 0.1 seconds required for this processing, and
The processing time is 0.133 seconds together with 033 seconds, and it is counted that two still images can be processed in that time.
Hereinafter, every time the video processing means (103) processes, the average video processing capability detection means (104) measures the processing time and the number of still images that can be processed. As in the case of 3, the process returns from the step 6 to the steps 1 and 3.

6 In step 2, after 5 to 0.033 seconds,
The original video output means (102) prepares the original video information, but the video processing means (103) is processing the video for 0.1 second in step 4 and cannot process the video. Using 501), still image information is stored.
Here, two pieces of still image information are accumulated (section B in FIG. 08).

After elapse of 0.1 second from 75, step 2
Then, since the video processing means (103) can be processed, the still image information stored first in the two still image information stored in step 6 is deleted from the read buffer and Output to means (103). On the other hand, the original video output means (102) outputs new original video information and stores it in the video buffering means (501). In the video buffering means (501), since one still image is deleted and one still image is accumulated, the accumulated amount does not change (FIG.
8 C section).

8. Here, another process running on the operating system enters a hibernation state to release computer resources.

9 In step 3, since the original video information has been input in step 7, the process proceeds to step 4, where the video processing means (10
In 3), the still image output in 7 is processed in 0.033 seconds and output as encoded video information. As in step 5, the process returns from step 6 to step 3.

10. Here, another process operating on the operating system enters the operating state again, and takes away a part of the computer resources.

After elapse of 0.033 seconds from 117, in step 2, since the video processing means (103) can be processed, the oldest still image information among the stored videos is read out and deleted from the buffer. Output to the video processing means (103). On the other hand, the original video output means (102) outputs new original video information and stores it in the video buffering means (501). In the video buffering means (501), since one still image has been deleted and one still image has been accumulated,
The amount of accumulation does not change.

12 In step 3, since the original image information was input in step 11, the process proceeds to step 4, where the image processing means
(103) processes the still image output in 7 in 0.066 seconds,
Output as encoded video information. Step 6 as in step 5
And returns to step 3.

13. In step 2, the original video output means (102) prepares the original video information 0.033 seconds after 11, but the video processing means (103) is processing the video for 0.066 seconds in step 12. , Because the video cannot be processed,
The still image information is stored using the video buffering means (501). Here, one still image information is stored (see FIG.
08 d section).

14. Similarly, after elapse of 0.066 seconds from 11, in step 2, since the video processing means (103) can be processed, the oldest still image information among the stored videos is read out and deleted from the buffer. And video processing means
Output to (103). In step 3, since the original video information has been input, the process proceeds to step 4, where the video processing means (103)
Processes this still image in 0.1 seconds. In the meantime, in step 2, three still images are output from the original video output means (102), but since one has been processed, two still images are conveniently stored in the video buffering means (FIG. 08). e section).

15 In step 6, the time here is 0.33
Seconds, the frame rate control time interval (10
Assuming that 8) has been reached, step 7 is executed. The average video processing capability detection means (104) has processed five still images so far, and the time required for processing is about 0.3
Since it was 3 seconds, the average processing time required to process a single still image is 0.0666..seconds, which is 15 fps when represented by the frame rate, so 15 fps is used as the video processing capability information (503). Output. Frame rate information output means (50
4) receives the video processing capability information (503) and outputs 15 fps as original video frame rate information (107). Upon receiving this, the original video output means sets the frame rate of the subsequent original video information to 15 fps. Further, the frame rate information output means (504) outputs a measurement reset signal (109) to the average video processing capability detection means (104). Return to step 3.

After 0.1 second has passed since 1614, in step 2, the video processing means (103) can be processed, so the oldest still image information among the stored videos is read out and deleted from the buffer. Output to the video processing means (103). In step 3, since there is input of the original video information, the process proceeds to step 4, where the video processing means (103) processes this still image in 0.033 seconds. Meanwhile, in step 2, 1
One still image is output from the original video output means (102), but since one has been processed, the storage amount of the video buffering means does not change (f section in FIG. 08).

After elapse of 0.033 seconds from 1716, in step 2, since the video processing means (103) can be processed, the oldest still image information among the stored videos is read out and deleted from the buffer. Output to the video processing means (103). In step 3, since there is input of the original video information, the process proceeds to step 4, where the video processing means (103) processes this still image in 0.1 second. Meanwhile, in step 2, 1
Since it was reset to 5 fps, one still image is output from the original video output means (102), but since one was processed,
The storage amount of the video buffering means does not change (g in FIG. 08).
section).

18 Similarly, the output of the original video information,
The process is repeated to obtain the h, i, and j sections in FIG. 08 as the storage amount of the video buffering means. The same operation is appropriately repeated until the end instruction is confirmed in step 1 (end).

As described above, in the video processing apparatus according to the present embodiment, the video processing capability detecting means detects the average frame rate of the still image information processed by the video processing means and outputs it as video processing capability information. The original video output means changes the frame rate of the output original video information according to the original video frame rate information obtained from the video processing capability information. FIG. 9 is a diagram for explaining the encoded video information obtained by this embodiment. In section A of FIG. 9, still images are arranged at 30 fps, but since the video processing means has only a processing capability of 15 fps, still images for section B are also stored in the buffer. At the beginning of section B, the frame rate of the original video information has been improved, and since then the amount of storage in the buffer has not increased, but in section B, the still images stored in the buffer up to that time are stored at 30 fps So, still images are lined up at 30fps. From section C, 15fp
Still images are lined up with s.

As described above, in the video processing apparatus according to the present embodiment, the frame rate of the still image is reset at a certain time interval in accordance with the fluctuation in the processing power of the video processing means, in accordance with the average of the processing power. I do. For this reason, in addition to the effect of obtaining an easy-to-view moving image without uneven dropouts described in the first embodiment, the frame rate is higher than that of the first embodiment in which the frame rate is adjusted to the worst value. There is an effect that an easy-to-view moving image can be obtained.

In the present embodiment, a buffer such as a FIFO is provided between the original video output means and the video processing means to buffer the data. Although a buffer is more effective and easy to realize, a buffer is not always necessary, and a desired effect can be obtained without a buffer depending on the configuration of the operating system. That is, the operation performed by the video processing means,
If the operations performed by other applications are distributed well at the timing when the frame enters, the desired effect can be achieved without a buffer.

In the present embodiment, since all still images cannot be processed on time at the originally set frame rate, an example in which the frame rate is reset to a lower value has been described. However, in this configuration example, it is obvious that the operation of returning the frame rate can be performed when other processes on the multitasking operating system are terminated and the video processing capability is improved. That is, when the average video processing capability returns from 15 fps to 30 fps again when the video processing capability is improved, the frame rate may be returned to 30 fps.

In the present embodiment, the processing time of each still image is measured, whereby the average video processing amount per unit time of the video processing means is measured. Alternatively, the method may be replaced with a method of measuring how many still images are processed by the video processing unit in a certain unit time. For example, in one example of the above-mentioned operation, since the video processing means has processed five pieces of still image information within 0.33 seconds, it is finally determined to be 15 fps, and the same result as described above is obtained. With this configuration, the video processing means has sufficient processing capability, but is particularly effective in a case where a still image is discarded due to video processing (maintaining a bit rate or the like). In other words, by counting only the processed still images without counting the discarded still images, the frame rate actually required by the video processing means can be determined for a certain period of time, and the original video output means is set to that frame rate. Then, the video processing means does not discard the still image, is efficient, and does not cause uneven drop of frames due to the convenience of the video processing means, so that a smooth moving image can be obtained. In the case of this configuration, if the setting of the bit rate of the coded video information is reset in the middle, a higher bit rate is required, and if it becomes necessary to restore the frame rate, the video processing means will do so. In such a case, a signal may be output to notify the user of an unexpected situation, or if the setting of the frame rate is constant for a prescribed time or more, the frame rate may be slightly increased to see the situation.

In this embodiment, the video processing capability information is directly used as the original video frame rate information.
As described in the first embodiment, the frame rate information output means may calculate and obtain the original video frame rate information based on the video processing capability information.

Further, in the present embodiment, the video processing capability detecting means measures the time spent for video processing by the video processing means, and thereby obtains the video processing capability.
Instead of such a direct method, it is also possible to adopt a configuration in which the speed of change in the amount of the original video information stored in the video buffering means is measured, and thereby the video processing capability is obtained indirectly. For example, if the image processing means is an existing product and the inside is a black box and the time spent for processing cannot be accurately measured, a method of indirectly obtaining the processing capacity is effective. Become. In this case, the speed at which the storage amount of the original video information increases indicates the difference between the frame rate of the original video information and the frame rate of the processing capability. Furthermore, when the video processing capability detection means does not have an accurate timer (when the operating system does not release the timer function to the application), it is inaccurate, but only the accumulated amount of the original video information Can be used to control the frame rate. In this case, the means for detecting the video processing capability is the same as the configuration shown in JP-A-5-64186, but in the configuration shown in the present embodiment, the original video information that may be discarded is buffered. Since it does not accumulate, in that respect,
Unlike the configuration shown in, there is an advantage.

In the present embodiment, the measurement reset signal is transmitted at every frame rate control time interval. However, it can be transmitted at other intervals. For example, if it is desired to measure the average processing time at a longer interval, the measurement reset signal may be transmitted at a longer time interval.

Further, in the flowchart of FIG. 7, the output of the original video information and the processing of the video are performed by multitasking. However, as shown in the first embodiment, this is an example, and Obviously, it may operate in the flow of the process.

In this embodiment, the average video processing capability detecting means outputs the video processing capability information in the form of a frame rate. However, this does not matter even in other forms (for example, the average processing time itself). It goes without saying.

(Embodiment 3) The video processing apparatus according to Embodiment 3 of the present invention does not always reset the frame rate in accordance with the video processing capability information, but uses the original video information stored in the video buffering means. By taking into account the accumulated amount of the image data and performing control such that the frame rate is not reset if the accumulated amount is small, encoded video information with more stable frames can be obtained.

FIG. 10 is a configuration diagram of a video processing apparatus according to the third embodiment of the present invention. In the figure, reference numeral 1001 denotes a frame rate information output unit according to the third embodiment. When outputting the original video frame rate information (107), not only the video processing capability information (503) but also the video buffering unit ( Reference is made to the original video buffer amount (1002) representing the amount of original video information stored in 501), and the original video frame rate information is output. The frame rate information output means (1001) in the present embodiment, when it is necessary to set the frame rate low, the original video buffer amount (1002)
If the amount does not exceed 10 still images, the value of the video processing capability information (503) is not output as it is as the original video frame rate information (107), but the original video frame rate information (107) is not output. ) And the video processing capability information (503), only half of the difference is subtracted from the current original video frame rate information (107). That is, the current original video frame rate information (107) is 30 fps, and the video processing capability information (503) is 15 fps.
If so, the new original video frame rate information (107) is 2
3 fps (22.5 fps is increased). Otherwise, the value of the shaku f image processing capability information (503) is output as it is as the original video frame rate information (107). The other reference numerals in FIG. 10 are the same as those in the second embodiment, and a description thereof will be omitted.

FIG. 11 is a diagram showing a flow of information in the video processing device according to the third embodiment.

In the figure, reference numeral 50102 denotes an original video buffer amount.
The details of this code are omitted since they have already been made in the description of FIG. Fig. 10 for other codes
Therefore, the description is omitted. FIG. 12 is a flowchart showing an example of the operation of the frame rate information output means (1001) of the video processing device according to the third embodiment. This flowchart shows a frame rate information output unit.
The operation of (1001) is focused on, and the operation of the other parts is the same as that of FIG. 7 shown in the second embodiment and is omitted. That is, FIG. 12 focuses on only steps 6 and 7 in FIG. Also,
FIG. 13 shows a video buffering unit according to the present embodiment.
This represents the transition of the amount of original video information stored in (501).

The operation of the video processing apparatus configured as described above will be described below with reference to FIGS. Since the entire operation except for the frame rate information output means (1001) is the same as that of the second embodiment, it will not be described here, and only the operation of the frame rate information output means (1001) will be described here.

1 In step 1 of FIG. 12, it is confirmed whether the current elapsed time has reached the frame rate control time interval (108). If the default value of 0.33 seconds has not been reached, step 2 is skipped in order to perform the next video processing. If the frame rate control time interval (108) has been reached, the process proceeds to step 2.

2. In step 2, the frame rate information output means (1001) checks the video processing capability information (503) in section A in FIG. 13 and obtains 15 fps. Then, the original video buffer amount (1002) is also confirmed, and it is obtained that five still images are accumulated. The frame rate setting must be reduced, but since it is less than 10 fixed images at a fixed value,
The original video frame rate information (107) is set to 23 fps instead of 15 fps. After that, the next video processing is performed.

3. When the process again arrived at step 2, although the video processing capacity was 15 fps in the section B in FIG. 13, the performance was improved in the section C and became 30 fps. That is, B section and C
Since the average video processing capability information (503) of the section is 23 fps, the current value of the original video frame rate information (107) is used as it is.

4. When the procedure again arrived at step 2, the video processing capacity remained at 30 fps in the section D in FIG. Since the original video frame rate information (107) was 23 fps, the original video buffer amount (1002) gradually decreased, leaving only one more frame. Since the video processing capacity in the D section was 30 fps, after that, the original video frame rate information (107) is 30 fps
Set to (end).

As described above, in the video processing apparatus according to the present embodiment, the frame rate information output means changes the degree to which the video processing capability information is considered according to the number of still images stored in the video buffering means. . In other words, a sudden start of another process may cause a sudden decrease in the video processing capability, and the phenomenon may immediately return to the original performance.
Even if the number of still images stored in the video buffering means is small, instead of immediately changing the original video frame rate information to 15 fps,
Instead of 15fps, it is also possible to process with a slightly larger value. That is, although it is preferable to prevent the occurrence of uneven frame intervals, it is not preferable that the frame rate greatly changes.
Even if the processing capacity suddenly drops to 15 fps, it will immediately reach 30 f
Since it returns to ps, it is possible to adopt a configuration in which the frame rate is not set to a very low value as long as it can be stored in the buffer.

In this embodiment, while the amount of data stored in the buffer is small, the frame rate is not set to be too large even if the video processing capability is lower than the frame rate of the original video information. there were. Conversely, when the amount stored in the buffer is large, even if the video processing capability is higher than the frame rate of the original video information, the frame rate is not set too high and stored in the buffer. A configuration in which control is performed in a direction to reduce the amount is possible.

Further, when still images are accumulated so that the buffer is likely to overflow, the frame rate of the original video information can be reduced even if the video processing capability is not reduced.

(Embodiment 4) The video processing apparatus according to Embodiment 4 of the present invention checks the free space of the buffer of the video buffering means, and thereby sets the frame rate control time interval. .

FIG. 14 is a configuration diagram of a video processing apparatus according to the fourth embodiment of the present invention. In the figure, reference numeral 1401 denotes a frame rate control time interval setting means, which checks a buffer free space (1402) of the video buffering means (501), and thereby sets a frame rate control time interval (108). is there. In the present embodiment, the frame rate control time interval setting means (1401) checks the acquired buffer capacity of the video buffering means before the video processing, and determines the amount and the initial value of the original video frame rate information. The frame rate control time interval (108) is set by the value and the amount of original video information per still image. Here, it is assumed that the initial value of the original video frame rate information is 30 fps, and the amount of the original video information per still image is 115200 bytes. Other symbols are the same as in the second embodiment. FIG. 15 is a diagram showing a flow of information in the video processing device according to the fourth embodiment. In the figure, reference numeral 1402 denotes a buffer free space, which indicates the free space of a buffer used by the video buffering means (501). The other reference numerals are the same as those in FIG. 14, and the description is omitted. further,
FIG. 16 is a flowchart illustrating an example of processing in the video processing device according to the fourth embodiment.

The operation of the video processing apparatus configured as described above will be described below with reference to FIGS.

1. Before actually performing video processing, in step 1 of FIG. 16, the frame rate control time interval setting means (501)
Confirms the capacity of the buffer used by the video buffering means (501) which has already been acquired, and confirms that the capacity is 2 Mbytes.

2 In step 2, the frame rate control time interval setting means (501) sets the buffer overflow even if the video processing means (103) does not operate at all due to some kind of failure after the video processing actually starts. The frame rate control time interval (108) is set so that the frame rate can be controlled without fail when the original video information has been stored to half the capacity of the buffer so that the control can be performed without causing an error. The original video frame rate information (107) is 30 fps, and the amount of one still image information is 11520 bytes. Therefore, when no video processing is performed, the time for storing the original video information to half of the buffer amount is about 0.3. Seconds.
Therefore, the frame rate control time interval (108) is set to 0.3 seconds (end).

As described above, in the video processing device of the present embodiment, the frame rate control time interval setting means (1401)
From the obtained buffer amount and the initial value of the frame rate, an interval for controlling the frame rate is calculated and obtained. The amount of buffer used by the video buffering means is
Since it changes depending on the amount of resources of the computer system on which the present configuration operates, and becomes larger or smaller, it is determined how often the frame rate is controlled based on the obtained buffer amount. If the amount of the buffer is sufficient, the frame rate can be controlled by accumulating the uncontrollable portion in the buffer, so that the frame rate does not need to be changed frequently, and as a result, the encoded video information whose frame rate changes gradually is easy to see. Can be obtained. If the amount of the buffer is small, the frame rate must be frequently controlled in order to prevent the buffer overflow and the drop of frames, and in this case, the frame rate fluctuates according to the change in the video processing capability. Thus, a coded video that is slightly difficult to see is obtained. Since the frame rate control interval thus depends on the size of the buffer, the frame rate control time interval setting means (1401) determines an optimal control interval in accordance with the buffer amount.

In the present embodiment, even if the video processing means does not function at all, the control interval is set so that the frame rate is controlled when half of the information in the buffer is accumulated. However, this is merely an example, and other calculation methods can be used.

In this embodiment, the frame rate control time interval setting means (1401) operates only once for setting the initial value before the actual video processing. During the process, the free space of the buffer may be monitored to thereby dynamically set the frame rate control interval. With this configuration, it is possible to prevent overflow of the buffer and to prevent occurrence of non-uniform dropout of frames even when the video processing performance fluctuates beyond the control interval set in advance. Furthermore, it goes without saying that the configuration shown in the present embodiment is effective even in combination with the first to third embodiments.

(Embodiment 5) The video processing apparatus according to Embodiment 5 of the present invention determines the performance of the apparatus itself before actually performing video processing, and according to the determination result, determines the frame rate of the original video information. Is obtained.

FIG. 17 is a configuration diagram of a video processing apparatus according to the fifth embodiment of the present invention. In the figure, reference numeral 1701 denotes an initial value setting means, which determines the performance of the apparatus itself, and calculates and outputs calculation capability information (1702) according to the result of the determination. In the present embodiment, the initial value setting means (1701) is a computer (CPU)
Of the original video information corresponding to the frequency is obtained using an internal correspondence table, and the output frame rate is output as calculation capability information (1702). Reference numeral 1703 denotes frame rate information output means according to the present embodiment.Before starting video processing, the original video frame rate information (107) is calculated according to the calculation capability information (1702) output by the initial value setting means (1701). It is to be obtained and output. In the present embodiment, the frame rate information output means (1703)
As the original video frame rate information (107)
Output. That is, if the calculation capability information (1702) is 30 fps, the initial value of the original video frame rate information (107) is also 30 fps. The other reference numerals are the same as those in the first embodiment, and a description thereof will be omitted. FIG. 18 is a diagram showing a flow of information in the video processing device according to the fifth embodiment. In the figure, reference numeral 1702 denotes video processing capability information. The details of this code are omitted since they have already been made in the description of FIG. Other reference numerals are the same as those in FIG. 1, and the description is omitted. FIG. 19 is a flowchart illustrating an example of processing in the video processing device according to the fifth embodiment. Further, FIG. 20 is a correspondence table showing the correspondence between the operating frequency of the computer and the calculation function information,
The initial value setting means (1701) is used at the time of initial value setting.

The operation of the video processing apparatus configured as described above will be described below with reference to FIGS. 17 to 20.

1 In step 1 of FIG. 19, initial value setting means
(1701) measures the operating frequency of the computer of the device by executing a simple benchmark test. Where 20
Check that it is operating at 0MHz.

2. In step 2, calculation capability information (1702) corresponding to the operating frequency is obtained by referring to the correspondence table shown in FIG. 20, and is output. Since the calculation capability information (1702) corresponding to 200 MHz was 30 fps, 30 fps is output.

3. In step 3, the frame rate information output means (1703) outputs the original video frame rate information (107) having the same value as the calculation capability information (1702) output in step 2.
The original video output means (102) sets a frame rate for outputting the original video information in the output original video frame rate information (107). As a result, the original video information is output at 30 fps.

(4) After step 3, the operation shown in the first or second embodiment is executed (end).

As described above, in the video processing apparatus according to the present embodiment, the initial value setting means (1701) measures the computer operating frequency of the apparatus, thereby obtaining the frame of the original video information suitable for the performance of the operating apparatus. Set the rate in advance. In the first and second embodiments, the configuration has been described in which, when the frame rate of the original video information is not suitable for the processing capability of the video processing means, the frame rate is reset to an optimal frame rate. Then, a frame rate that is deemed appropriate for the processing capability is set in advance. For this reason, there is an effect that the optimum frame rate can be quickly set when the actual video processing is performed.

In the present embodiment, the calculation capability information is directly used as the original video frame rate information.
The frame rate information output means may calculate and obtain the original video frame rate information based on the calculation capability information. For example, the original video frame rate information may be obtained based on the calculation function information in consideration of a preset parameter for performing video processing.

Further, in this embodiment, the method in which the initial value setting means performs a simple benchmark test and thereby predicts the operating frequency of the device has been described. Read the information indicating the model number of the device, read the information of other processes that are running at that time, find the computer resources that can be used by the video processing process, and actually execute the video processing on a test basis. The frame rate of the processed original video information may be used as the calculation capability information. Furthermore, it goes without saying that the configuration shown in the present embodiment is effective even in combination with the first to fourth embodiments.

Further, the first to fifth embodiments described so far.
Has been described as an example of video processing, that is, encoding of video. However, this is merely an example, and other processing can be performed. For example, the present invention can be applied to a case where a moving image analysis of an image captured in real time is performed, and a case where color tone conversion is performed on a moving image captured in real time.

Further, the method of the present invention described so far is
If it is realized as a program product and stored and distributed on a floppy disk or other storage medium,
It is needless to say that the present invention is effective even when distributed through a medium such as a computer network.

[0149]

According to the first aspect of the present invention, when the original image information including a plurality of still image information representing still images at each still image time interval, which is an arbitrary time interval, is output, the original image Video processing means for processing information, and original video frame rate information indicating the number of the still images per unit time at every frame rate control time interval which is an arbitrary time interval according to the degree of progress of the processing of the video processing means. Frame rate information output means for obtaining and outputting, and an original image for outputting the original image information and changing a still image time interval of the original image information according to the original image frame rate information when the original image frame rate information is output With the provision of output means, the frame rate of the original video information can be reset according to the video processing capability. Mau prevents spend wasteful computing power for output of the original image information, it is possible to obtain a smooth processed video information interrupted video not more evenly does not occur.

According to a second aspect of the present invention, in the video processing apparatus according to the first aspect, the video processing means obtains a processing time for processing one piece of still image information, and the measurement reset signal is included in the processing time. Maximum video processing time detection means for obtaining and outputting video processing capability information from the maximum value until the maximum value is transmitted, and frame rate information output means, transmits the measurement reset signal at any time interval, The above effect can be obtained by obtaining and outputting the original video frame rate information in accordance with the video processing capability information output by the maximum video processing time detecting means at every frame rate control time interval.

According to a third aspect of the present invention, in the video processing apparatus of the first aspect, the average processing amount of the original video information per unit time of the video processing means in the section until the measurement reset signal is transmitted. The average video processing capability detection means for obtaining and outputting video processing capability information from the average processing amount and the frame rate information output means transmit the measurement reset signal at an arbitrary time interval. The above-mentioned effects are obtained after obtaining a higher quality video processing result by obtaining and outputting the original video frame rate information in accordance with the video processing capability information output by the average video processing capability detecting means. be able to. In addition, there is an effect that it is possible to prevent occurrence of uneven frame drop due to the convenience of video processing.

According to a fourth aspect of the present invention, in the video processing apparatus of the first, second or third aspect, when the video processing means cannot process the original video information, the original video information is temporarily stored. When the video processing means can process the original video information, the video processing means includes a video buffering means for outputting the original video information accumulated most recently to the video processing means, so that a sudden video The above effects can be more effectively obtained when the processing capacity changes.

According to a fifth aspect of the present invention, in the video processing apparatus according to the fourth aspect, the maximum video processing time detecting means or the average video processing capability detecting means is adapted to store the original video information stored in the video buffering means. Since the video processing capability information is obtained and output according to the change in the amount, the above-described effect can be obtained even in an environment where the processing time spent for video processing cannot be accurately measured.

According to a sixth aspect of the present invention, in the video processing apparatus according to the fourth aspect, when the frame rate information output means determines the original video frame rate information, the frame rate information output means includes not only the video processing capability information but also the video buffering means. The original video frame rate information is obtained and output according to the storage amount of the original video information stored in the camera, so that the change in the frame rate is less and the buffer overflow can be prevented. Can be obtained.

A video processing apparatus according to a seventh aspect of the present invention is the video processing apparatus according to the fourth aspect, further comprising a frame rate control time interval setting means for changing the frame rate control time interval in accordance with the free space of the buffer of the video buffering means. With this configuration, buffer overflow can be completely prevented, and the above effects can be obtained effectively.

An image processing apparatus according to claim 8 is the video processing apparatus according to any one of claims 1 to 7, wherein the performance of the apparatus itself is determined, and in accordance with the result of the determination, calculation capability information is obtained and output. The value setting means and the frame rate information output means determine and output the original video frame rate information according to the computational capability information output by the initial value setting means before the video processing means starts the video processing, Video processing can be performed at an optimal frame rate from the time when the video processing is started, and the above effects can be obtained more effectively.

In the video processing method according to the ninth aspect, when original video information including a plurality of still image information representing still images at still image time intervals, which are arbitrary time intervals, is output,
An image processing step of processing the original image information, and an original image frame representing the number of the still images per unit time for each frame rate control time interval that is an arbitrary time interval according to the degree of progress of the processing of the image processing step A frame rate information output step of obtaining and outputting rate information; and changing the still image time interval of the original video information according to the original video frame rate information when the original video information is output and the original video frame rate information is output. And the original video output step, which enables the frame rate of the original video information to be reset according to the video processing capability. Obtaining smooth processed video information that prevents unnecessary computational power and prevents uneven video breaks It can be.

According to a tenth aspect of the present invention, in the video processing method according to the ninth aspect, the video processing step obtains a processing time for processing one piece of still image information, and the measurement reset signal is included in the processing time. A maximum video processing time detection step of obtaining and outputting video processing capability information from the maximum value until a signal is transmitted, and a frame rate information output step transmits the measurement reset signal at an arbitrary time interval, The above effect can be obtained by obtaining and outputting the original video frame rate information in accordance with the video processing capability information output by the maximum video processing time detection step at each frame rate control time interval.

An image processing method according to an eleventh aspect of the present invention is the video processing method according to the ninth aspect, wherein the average processing amount of the original image information per unit time of the image processing step in the section until the measurement reset signal is transmitted. The average video processing capability detection step of obtaining and outputting the video processing capability information from the average processing amount obtained, and the frame rate information output step transmits the measurement reset signal at an arbitrary time interval,
Also, by obtaining and outputting the original video frame rate information in accordance with the video processing capability information output by the average video processing capability detection step at each frame rate control time interval, a higher quality video processing result was obtained. Above, the above effects can be obtained. In addition, there is an effect that it is possible to prevent occurrence of uneven frame drop due to the convenience of video processing.

According to a twelfth aspect of the present invention, in the video processing method of the ninth, tenth or eleventh aspect, when the video processing step cannot process the original video information, the original video information is temporarily stored. When the video processing step is capable of processing the original video information, the video processing step includes a video buffering step of outputting the previously stored original video information to the video processing step. The above effects can be more effectively obtained when the processing capacity changes.

According to a thirteenth aspect of the present invention, in the video processing method according to the twelfth aspect, the step of detecting the maximum video processing time or the step of detecting the average video processing capability includes the step of detecting the original video information stored in the video buffering step. Since the video processing capability information is obtained and output according to the change in the amount, the above-described effect can be obtained even in an environment where the processing time spent for video processing cannot be accurately measured.

According to a fourteenth aspect of the present invention, in the video processing method according to the twelfth aspect, the frame rate information output step includes, when obtaining the original video frame rate information, not only the video processing capability information but also the video buffering step. By obtaining and outputting the original video frame rate information according to the storage amount of the original video information stored in the
The change in the frame rate is less, the buffer overflow can be prevented, and the above effects can be obtained effectively.

A video processing method according to a fifteenth aspect of the present invention is the video processing method according to the twelfth aspect, further comprising a frame rate control time interval setting step of changing a frame rate control time interval according to a free space in a buffer in the video buffering step. With this configuration, buffer overflow can be completely prevented, and the above effects can be obtained effectively.

According to a sixteenth aspect of the present invention, in the video processing method according to any one of the ninth to fifteenth aspects, the performance of the operating device itself is determined, and calculation capability information is obtained and output according to the result of the determination. The initial value setting step and the frame rate information output step determine and output the original video frame rate information in accordance with the calculation capability information output by the initial value setting step before the video processing step starts the video processing. Thus, the video processing can be performed at the optimum frame rate from the time when the video processing is started, and the above effects can be obtained more effectively.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a video processing device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a flow of information of the video processing device according to the first embodiment of the present invention.

FIG. 3 is a flowchart of the video processing device according to the first embodiment of the present invention;

FIG. 4 is a diagram showing a configuration of original video information processed according to the first embodiment of the present invention, and finally obtained encoded video information;

FIG. 5 is a configuration diagram of a video processing device according to a second embodiment of the present invention.

FIG. 6 is a diagram illustrating a flow of information of the video processing device according to the second embodiment of the present invention.

FIG. 7 is a flowchart of the video processing device according to the second embodiment of the present invention;

FIG. 8 is a graph showing a change in the amount of original video information stored in a video buffering unit according to the second embodiment of the present invention;

FIG. 9 is a diagram illustrating a configuration of encoded video information finally obtained according to the second embodiment of the present invention;

FIG. 10 is a configuration diagram of a video processing device according to a third embodiment of the present invention.

FIG. 11 is a diagram illustrating a flow of information of the video processing device according to the third embodiment of the present invention.

FIG. 12 is a flowchart illustrating an operation of a frame rate information output unit of the video processing device according to the third embodiment of the present invention;

FIG. 13 is a graph showing an example of a change in the amount of information stored in the video buffering unit according to the third embodiment of the present invention.

FIG. 14 is a configuration diagram of a video processing device according to a fourth embodiment of the present invention.

FIG. 15 is a diagram illustrating a flow of information of the video processing device according to the fourth embodiment of the present invention.

FIG. 16 is a flowchart illustrating an operation of a control interval setting unit of the video processing device according to the fourth embodiment of the present invention;

FIG. 17 is a configuration diagram of a video processing device according to a fifth embodiment of the present invention.

FIG. 18 is a diagram illustrating a flow of information of the video processing device according to the fifth embodiment of the present invention.

FIG. 19 is a flowchart of the video processing device according to the fifth embodiment of the present invention.

FIG. 20 is a diagram illustrating an example of a correspondence table used by the initial value setting unit according to the fifth embodiment of the present invention to obtain calculation capability information;

FIG. 21 is a configuration diagram of a video processing device according to a conventional technique.

FIG. 22 is a view for explaining the flow of information in a video processing apparatus according to a conventional technique.

FIG. 23 is a flowchart of a video processing apparatus according to a conventional technique.

FIG. 24 is a diagram showing a configuration of encoded video information obtained by a video processing device according to a conventional technique.

FIG. 25 is a configuration diagram of a video processing device according to Prior Art 1.

FIG. 26 is a view for explaining the flow of information of a video processing device according to Prior Art 1;

FIG. 27 is a flowchart of a video processing device according to Prior Art 1;

[Explanation of symbols]

 Reference Signs List 101 video camera 102 original video output unit 103 video processing unit 104 video processing capability detection unit 105 video processing capability information 106 frame rate information output unit 107 original video frame rate information

──────────────────────────────────────────────────の Continued on front page (51) Int.Cl. ⁶ Identification code FI H04N 5/781

Claims (16)

[Claims] 1. An image processing means for processing original image information, when original image information comprising a plurality of still image information representing still images for each still image time interval, which is an arbitrary time interval, is output. Frame rate information output means for obtaining and outputting original video frame rate information representing the number of still images per unit time for each frame rate control time interval which is an arbitrary time interval according to the degree of progress of the processing of the video processing means, Original video output means for outputting the original video information and changing a still image time interval of the original video information in accordance with the original video frame rate information when the original video frame rate information is output, Video processing device. 2. A video processing apparatus according to claim 1, wherein said video processing means obtains a processing time for processing one piece of still image information, and determines a maximum value of said processing time until a measurement reset signal is transmitted. The maximum video processing time detecting means for obtaining and outputting the video processing capability information from the obtained maximum value, and the frame rate information output means transmit the measurement reset signal at an arbitrary time interval, and at every frame rate control time interval. An image processing apparatus, wherein original image frame rate information is obtained and output according to image processing capability information output by the maximum image processing time detecting means. 3. The video processing apparatus according to claim 1, wherein an average processing amount of the original video information per unit time of the video processing means in a section until the measurement reset signal is transmitted is determined. Means for detecting and outputting capability information, and means for outputting frame rate information, wherein the means for outputting frame rate information transmits the measurement reset signal at arbitrary time intervals, and the means for detecting average image processing capability at each frame rate control time interval. A video processing apparatus characterized in that original video frame rate information is obtained and output in accordance with video processing capability information output by the video processing device. 4. The video processing apparatus according to claim 1, wherein said video processing means temporarily stores said original video information when said video processing means cannot process said original video information, and said video processing means stores said original video information in said video processing means. An image processing apparatus comprising: an image buffering unit that outputs the original image information accumulated most recently to the image processing unit when image information can be processed. 5. The video processing apparatus according to claim 4, wherein the maximum video processing time detecting means or the average video processing capacity detecting means is adapted to change the amount of the original video information stored in the video buffering means. A video processing apparatus characterized in that a video signal is obtained and output. 6. The video processing apparatus according to claim 4, wherein the frame rate information output means is configured to output not only the video processing capability information but also the original video information stored in the video buffering means when obtaining the original video frame rate information. An image processing apparatus characterized in that original image frame rate information is obtained and output according to a storage amount. 7. The video processing apparatus according to claim 4, further comprising a frame rate control time interval setting means for changing a frame rate control time interval according to a free space in a buffer of the video buffering means. apparatus. 8. An image processing apparatus according to claim 1, wherein said apparatus determines the performance of said apparatus itself, and calculates and outputs calculation capability information in accordance with a result of said determination. A video processing apparatus, wherein the output means obtains and outputs original video frame rate information in accordance with the computing capability information output by the initial value setting means before the video processing means starts video processing. 9. An image processing step of processing original image information when original image information comprising a plurality of pieces of still image information representing still images for each still image time interval that is an arbitrary time interval is output; A frame rate information output step of obtaining and outputting original video frame rate information representing the number of still images per unit time for each frame rate control time interval that is an arbitrary time interval according to the degree of progress of the processing of the video processing step, Outputting the original video information and, when the original video frame rate information is output, changing a still image time interval of the original video information according to the original video frame rate information. Video processing method. 10. The video processing method according to claim 9, wherein
The maximum image for which the image processing step obtains the processing time for processing one piece of still image information, obtains the maximum value of the processing time until the measurement reset signal is transmitted, obtains the image processing capability information from the maximum value, and outputs it. The processing time detecting step and the frame rate information outputting step transmit the measurement reset signal at an arbitrary time interval, and according to the video processing capability information output by the maximum video processing time detecting step at every frame rate control time interval. A video processing method characterized by obtaining and outputting original video frame rate information. 11. The video processing method according to claim 9, wherein
An average video processing capability detection step of obtaining an average processing volume of original video information per unit time of a video processing step in a section until a measurement reset signal is transmitted, obtaining video processing capability information from the average processing volume, and outputting the frame; The rate information output step transmits the measurement reset signal at an arbitrary time interval, and obtains original video frame rate information according to the video processing capability information output by the average video processing capability detection step at each frame rate control time interval. A video processing method characterized by outputting. 12. The video processing method according to claim 9, 10 or 11, wherein when the video processing step cannot process the original video information, the original video information is temporarily stored, and the video processing step is performed by the video processing step. A video buffering step of outputting the oldest stored original video information to the video processing step when the video information can be processed. 13. The video processing method according to claim 12, wherein the maximum video processing time detecting step or the average video processing capacity detecting step is performed in accordance with a change in the amount of original video information stored in the video buffering step. And outputting the image. 14. A video processing method according to claim 12, wherein the step of outputting frame rate information includes the step of obtaining not only video processing capability information but also original video information accumulated in the video buffering step when obtaining the original video frame rate information. A video processing method comprising: obtaining and outputting original video frame rate information according to a storage amount. 15. A video processing method according to claim 12, further comprising a frame rate control time interval setting step of changing a frame rate control time interval according to a free space of a buffer in the video buffering step. Method. 16. An image processing method according to claim 9, wherein the performance of the operating device itself is determined, and an initial value setting step of obtaining and outputting calculation capability information according to the result of the determination; A video processing method, wherein the rate information output step obtains and outputs original video frame rate information in accordance with the computational function information output by the initial value setting step before the video processing step starts video processing.