JPH05165931A - Image data processing system - Google Patents

Abstract

PURPOSE:To hourly exactly reproduce image data without the sense of incongruity and to reproduce the image data even by the computer of low performance. CONSTITUTION:A division processing means 21 divides the respective processings of image data as processing units so as to complete the processings within time allowed to schedule time corresponding to the processing contents of the image data, and a reproducing part 22 calls a time control means 31 each time the processing is completed for each processing unit. Then, the time control means 31 executes the processings in an order scheduled by the call from the reproducing part 22 and when the reproducing time due to the reproducing part 22 is delayed rather than the schedule time, the processing is omitted for each processing unit corresponding to the delay time. When the reproducing time is advanced rather than the schedule time, the processing is on standby.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention can reproduce image data in a computer system for processing image data such as music and moving images in real time, accurately and temporally without any audible or visual discomfort. The present invention relates to an image data processing method, which enables even computer systems having different processing performances to reproduce the same image data with a quality that can be realized by the system.

[0002]

2. Description of the Related Art Ordinary real-time processing or real-time processing means that a certain processing is requested and the time until the processing for this request is started is very short.

On the other hand, the time delay (hereinafter referred to as the required time accuracy) that is permissible for the reproduction of image data handled in music or moving images is 1-2 ms at the maximum for sound.
The maximum is about 50 milliseconds for an image.

However, the conventional general-purpose computer system has not been a system that can guarantee the time accuracy required for reproducing music and moving images. Therefore, considerable system performance is required to satisfy the time accuracy using the conventional computer system.

On the other hand, the electronic exchange carries out processing with the above-mentioned required time accuracy. This is because the electronic exchange is premised on that each processing is relatively simple and the time required for the processing is stable.

On the other hand, since the time required to reproduce the image data varies, this processing method cannot be applied as it is. For the above reasons, in order to reproduce and process conventional image data, a hardware architecture suitable for it, such as a game machine, should be added, or dedicated hardware should be added to a general-purpose computer system, or at worst. Measures were taken such as using a high-performance system so that the required performance can be obtained even at times.

[0007]

However, in the processing method using the dedicated hardware architecture, not only does the cost increase, but even if the performance of the entire system improves, the performance of the key dedicated hardware does not improve. ,
The quality obtained when reproducing the image data is not improved.

Further, a system which can provide the required performance even in the worst case is generally expensive. As a result, there is a problem that the system becomes much more expensive than that in order to obtain the quality that can be realized by the inexpensive game machine or the dedicated player.

Further, according to the program scheduling method in the conventional system, the process being executed is interrupted and interrupted (hereinafter, referred to as "interruption") at a certain time interval (time slice) set independently of the data and the processing content to be processed by the system. It was called preemption.), And often required processing was performed at that time. In this case, since the fixed time slice is not the time unit perceived by humans, the reproduced image data feels unnatural.

An object of the present invention is to reproduce image data accurately and audibly or visually without any discomfort.
An object of the present invention is to provide an image data processing method capable of reproducing the same image data with a quality that can be realized by the system even in a system having different processing performance.

[0011]

The present invention has a constitution shown in FIG. 1 as a first invention in order to solve the above problems and achieve the object. The present invention is a computer system that reproduces image data 1 including image data, sound data, and data for a reproduction order of these data and a schedule time to be reproduced. A division processing unit 21 that divides the processing so that the processing is completed within an allowable time with respect to the schedule time according to the processing content of the data and marks each divided portion as a processing unit, and the division processing unit 21. Playback unit 22 for playing back image data for each processing unit divided by
And a time control means 31 for performing the processing in a scheduled order based on the reproduction order and schedule time data in the image data and the marks provided for each processing unit from the division processing means 21.

The reproducing section 22 calls the time control means 31 every time the processing of each processing unit is completed, and the time control means 3 is called.
1 performs processing in a scheduled order by calling from the reproduction unit 22, and when the reproduction time by the reproduction unit 22 lags behind the schedule time, the processing is omitted in processing units according to the delay time, and the reproduction time exceeds the schedule time. When proceeding, it is configured to wait for processing.

[0013]

According to the present invention, each processing of the image data is divided by the division processing means into processing units according to the processing contents of the image data, and the time is set every time the reproduction unit finishes the processing of the processing units. The control means is called, the time control means performs the processing in the order scheduled using the reproduction order and the schedule time data, and when the reproduction time is delayed from the schedule time, the processing is omitted in processing units according to the delay time,
When the reproduction time advances from the scheduled time, the process waits, so that the reproduction speed can be adjusted.

Therefore, the reproduced image data can be processed even if the reproduced image data is relatively natural and cannot be reproduced by a computer system having a low performance. The image data is a moving image such as an animation or a movie and sounds and music accompanying it. The image data targeted by the present invention is (a) the time accuracy for simultaneously showing two image data of the same type is 1 to 2 milliseconds for sound and 20 to 5 for image.
A time difference of about 0 milliseconds is allowed. (B) On the other hand, the time interval actually perceived as a meaningful unit is longer. (C) Even if a certain block of image data is omitted, it does not affect subsequent processing. Has the characteristic.

In the state assumed by the present invention, it is possible that a large number of such requests continue to occur and the previous request cannot be processed by the next request. Actually, the time accuracy described in (a) is allowed only at the breaks in the image data, and is not allowed when arbitrarily divided.

Further, depending on the image data, there is a characteristic that if the division is done properly, even if one unit in the middle is omitted, it does not affect the overall feeling. For example, double-speed reproduction of video or the like utilizes this characteristic.

The image data may be data composed of a plurality of frames. Furthermore, the time when each process is completed is, for example, a break in the sound effect of a certain mass of people, a place where breathing of music or a pause symbol, or, in the case of an image, a time corresponding to a horizontal / vertical synchronizing signal.

Further, the image coordinates are changed for each processing unit,
You can specify the color change and change the volume and pitch of the sound.

[0019]

EXAMPLES Specific examples of the present invention will be described below.
FIG. 2 is a diagram showing a storage form of image data processed in the embodiment of the present invention. Here, the image data is a moving image such as an animation or a movie, and sound and music accompanying it. First, the characteristics of this image data will be described below.

Regarding the sound, the time accuracy (temporal error) in reproducing the sound PCM-sampled by a simple calculation influences the S / N ratio at almost the same ratio. That is, for example, if the strain is suppressed to about 0.1%,
Time accuracy is also required. For example, 1KH
When reproducing the z signal (sampling rate is 2 K / sec or more), time precision of 0.5 microsecond is required. When discussing signal quality in this way, high accuracy is required.

Further, according to a psychological experiment, when two visual (light) stimuli are given to a human at a certain time difference, if they are within a certain time difference, they feel that they are simultaneous, but if they are exceeded, one It has been found that there are thresholds at which the stimulus source feels moved.

This threshold is to recognize one stimulus,
Long enough, but short enough to recognize meaningful movements. Although it depends on how to give the stimulus, it is considered to be about 50 milliseconds.

The same applies to sounds. In other words, when two sounds of the same kind are given as stimuli, there is a time unit that should be called a phoneme with a time interval of 1 to several milliseconds.
A delay within this time interval does not feel a time difference, but if the delay becomes larger than this, it feels that the sound has changed or that there are two sound sources. Note that this time interval corresponds to a time of about one wavelength when the human voice frequency is about 100 to 3 KHz.

As in the case of light, it takes a longer time of about tens of milliseconds to reorganize sounds as meaningful units (eg, chords). Also, as we experience everyday, just because we miss a sound or overlook an image does not mean that we cannot understand the sound and image after that.

Therefore, the image data targeted by this embodiment requires (1) high time accuracy when the quality (S / N) of each image is a problem. (2) The time accuracy for showing two image data of the same kind at the same time is 1 to 2 milliseconds for sound and 20 to 5 for image.
A time difference of about 0 milliseconds is allowed. (3) On the other hand, the time interval actually perceived as a meaningful unit is longer. (4) Even if a certain block of image data is omitted, it does not affect subsequent processing. Has the characteristic.

The time accuracy required at the level (1) is extremely high, and under the present circumstances, it is preferable to use a dedicated circuit / hardware. Further, the accuracy required by (2) can be targeted in the conventional computer system as the time accuracy, but in the state assumed in the present embodiment, a large number of such requests continue to occur, It is possible that the previous request cannot be processed by the next request.

The time accuracy described in (2) is actually allowed only at the breaks in the image data, and is not allowed at the time of arbitrary division. It is necessary to note that.

Furthermore, depending on the image data, there is a property that if it is divided properly, even if one unit in the middle is omitted, it does not affect the overall feeling. Double-speed playback of video etc. uses this property.

Next, as shown in FIG. 2, the image data has a plurality of blocks BK each consisting of one frame of image data, sound effect data, and music (for example, BGM data), and each block BK contains an internal block. A control block CB in which the reproduction order is described for the above data is added as appropriate.

The image data is stored in an image data storage unit (not shown) in a format that can be processed in advance in a row unit (one raster unit) as shown in FIG. 2 according to the display screen format (raster scan). Has been done. That is, the image data storage unit schedules each processing of the image data 1 including the image data, the sound data, the reproduction order of these data, and the schedule time data to be reproduced according to the processing content of the image data 1. Image data is stored in processing units (one raster unit) by dividing the processing into small portions so that the processing is completed within the time allowed for the time.

The following specifications are described in the control block CB. The playback time of the image data is
The absolute time from the start of the entire reproduction or the relative time from the previous image is shown.

The time at which a certain sound effect data is reproduced is indicated by the corresponding image data or absolute time.
The reproduction start time and end time of the BGM data are indicated by the corresponding image data or absolute time.

Further, the reproduction order of the image data and BGM data can be designated, and interlacing and repetition can be described. Unless otherwise specified in the control block CB, it is assumed that the image data are arranged in time series.

FIG. 3 is a block diagram showing the arrangement of an image data reproducing section for reproducing image data. In the embodiment, animation reproduction is performed. The image data reproducing unit 2 includes a preprocessing unit 21, an image reproducing unit 22, and B.
GM reproducing unit 23, sound effect reproducing unit 24, scheduler 31
have.

The pre-processing section 21 creates data to be output from the image data 1 to the respective reproduction sections, marks each processing unit based on the image data 1, and indicates the reproduction time of the control block CB and the music / sound effect. These are subdivided by the marks described in the data and output as an instruction to the scheduler 31.

Here, the mark is a mark attached to a time point at which a process being executed at a certain fixed time interval can be interrupted by an interrupt or the like, that is, a time point at which preemption is possible. The preemptible time is, for example, a break in the sound effect of a certain crowd, a place of breath or pause symbols in music, or an image,
This is the time corresponding to the horizontal / vertical sync signal.

The image reproducing section 22 reproduces the image data processed by the preprocessing section 21. BGM playback unit 2
3 is for reproducing BGM data, and the sound effect reproducing section 24 is for reproducing sound effect data.

The scheduler 31 controls the control
The control table 26 in which the reproduction order and schedule time data from the block CB and the mark for each processing unit are registered, and the timer 27 are provided, and each processing (reproduction) is performed in the order scheduled based on the control table 26. When the reproduction time of the image data from the image reproduction unit 22 is behind the schedule time read from the control table 26, the processing is omitted in processing units according to the delay time by the timer 27.

FIG. 4 is a diagram showing the time required when the image data compressed and stored on the personal computer is reproduced. FIG. 4 shows the time required when reproducing data of 200 to 400 lines. As a result, the time required to reproduce one line is 1 millisecond or less.

FIG. 5 is a diagram showing the flow of processing while displaying one screen. The image reproducing unit 22 processes the image data of one frame (one screen) for each line (one raster), and calls the scheduler 31 each time the processing for one line is completed. If there is sound / BGM data to be reproduced within 1 millisecond before and after the end of processing for one line, BG
The M reproducing unit 23 reproduces a block of sound / BGM data, sends the BGM data to the sound source block 25, and outputs the BGM. Here, since the BGM data is reproduced in units of 1 mm or less in association with the reproduction of one line of image data, the sound feels natural. If there is no sound / BGM data to be reproduced, the next image data is reproduced.

Of course, when there is sound effect data to be reproduced within 1 millisecond before and after the end of the processing for one line, the sound effect reproducing section 24 can reproduce the sound effect data. Next, when the image data of the next frame is reproduced, the reproduction time of the image data from the image reproduction unit 22 and the schedule time from the control table 26 in the scheduler 31 are compared by the scheduler 31, and the schedule time is calculated. Also, when the reproduction time of the image data is delayed, the scheduler 31 performs processing according to the degree of delay (in the case shown in FIG. 4, the time difference between frames is 1).
0-200 ms or 50-450 ms, 5
If it is not less than 0 milliseconds, the image looks unnatural. ). (1) For example, when the delay is within one frame (one frame), the image reproduction unit 22 reproduces the odd-numbered row in the odd-numbered image by the control signal from the scheduler 31, regardless of the original image. , In the even-numbered image, the even-numbered image is reproduced. (2) When the delay is one frame or more, the image reproducing unit 22 reproduces the image by skipping one frame by the control signal.

By performing thinning (omission) of images according to the degree of delay in this way, the playback speed can be adjusted, and the playback time by each playback unit can be synchronized with the schedule time.

On the other hand, when the reproduction time of the image data is ahead of the scheduled time, the process waits until the scheduled time. As a result, the following effects are obtained. (1) Since the image data that matches the human sense characteristics is reproduced, the reproduced result feels relatively natural. The reproduction is more natural than the mechanically measured numbers such as the S / N ratio and the strain rate. (2) Even in a computer system having low performance, image data that cannot be reproduced by the conventional method can be reproduced. For example, consider the case of reproducing image data that requires about 20% higher performance for a computer system with relatively lower performance.

According to the conventional method, the whole is uniformly 20%.
Delayed or the sound (image) is processed as requested,
The image (sound) is reproduced 20% to 30% later than the original time. In either case, the image data will be different from what it originally represents.

On the other hand, according to the method of this embodiment, each of the sound, the image, etc. is divided into small units, and each is reproduced at the original time. At this time, about 20% of the entire image data is lost. However, normally, even if about 20% of image data is missing, there is no particular problem because the contents that should be displayed can be displayed. (3) There is little overhead associated with the schedule. Since the time when the processing is preempted can be predicted, the overhead required for register saving and resource conflict avoidance does not occur as compared with the conventional method such as an interrupt. This overhead cannot be ignored when processing is performed in a short time such as 1 to 2 milliseconds.

As described above, according to this embodiment, by setting the schedule unit of the program as a processing unit adapted to the time unit perceived by humans, it is more natural than that obtained by the conventional method, Flexible image playback is possible.

[0047]

According to the present invention, the program schedule unit is a processing unit adapted to the time unit perceived by humans, so that the reproduction of the image data suitable for the human sense characteristics is performed. , The result of reproduction is relatively natural. Further, even in a computer system having low performance, image data that cannot be reproduced by the conventional method can be reproduced. Furthermore, the overhead associated with the schedule is small.

[Brief description of drawings]

FIG. 1 is a principle diagram of the present invention.

FIG. 2 is a diagram showing how image data used in the embodiment is stored.

FIG. 3 is a configuration block diagram of an image data reproducing unit.

FIG. 4 is a diagram showing a time required for reproducing image data.

FIG. 5 is a diagram showing a flow of processing while displaying one screen.

[Explanation of symbols]

 1 ... Image data 2 ... Image data playback unit 21 ... Pre-processing unit 22 ... Image playback unit 23 ... BGM playback unit 24 ... Sound effect playback unit 25 ... Scheduler BK ... Block CB ... Control ... block

Claims (1)

[Claims] 1. A computer system for reproducing image data (1) including image data, sound data and data for a reproduction order of these data and a schedule time to be reproduced, wherein each of the image data is reproduced. A division processing unit (2) that divides the processing so that the processing is completed within a time allowed for the schedule time according to the processing content of the image data and marks each divided portion as a processing unit.
1), a reproduction unit (22) for reproducing image data for each processing unit divided by the division processing unit (21), a reproduction order and schedule time data in the image data, and the division processing unit (21). And a time control means (31) for performing the processing in a scheduled order based on the marks attached to the respective processing units, and the reproducing section (22), each time the processing of each processing unit is completed, The time control means (31) is called, and the time control means (31) performs the processing in the scheduled order according to the call from the reproduction section (22), and the reproduction time by the reproduction section (22) is the schedule. Image data characterized by omitting processing in the processing unit according to the delay time when the time is later than the time, and waiting when the reproduction time is earlier than the scheduled time Processing method.