JP4401861B2 - Image processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus and a control method thereof, and is applied to, for example, an image distribution system that distributes a video stream using a network.

  In recent years, the Internet has become widespread, and information transmission by WWW (World Wide Web) or the like has been generally performed. Under such circumstances, systems have emerged that have the function of shooting video and transmitting the video to the network in real time.

  On the other hand, various image compression methods are used for distributing images using a network. For example, as a still image compression method widely used, there is JPEG (Joint Photograph coding Experts Group). As a moving image compression method, there are M-JPEG (Motion-JPEG) in which this JPEG method is applied to moving image distribution, MPEG2 and MPEG4 (Moving Picture Experts Group phase 2/4), and the like. Of these methods, which method is used to distribute an image can be variously considered depending on the system configuration on the video distribution side, the user's selection, and the like.

  Furthermore, recently, a plurality of types of communication lines having various data transfer rates have been provided. Also, the terminals for viewing the distributed video vary from those having a relatively small screen of the cellular phone class, those using a part of the display screen using a PC, and those using a dedicated terminal. The side that constructs a system for distributing image data is required to provide services in response to the various demands of these users. In addition, a network camera server that realizes remote camera operation is required to have real time and low delay.

  Here, as a first example of a conventional video distribution system having a mechanism for providing a service in response to various requests of a user, an image processing apparatus 200 as shown in FIG. 17 is provided. There is something that was done.

  In the image processing apparatus 200 as shown in FIG. 17, image data supplied from a video camera or the like is given to each of the image processing units 210a, 210b, and 210c in units of frames. Each of the image processing units 210a, 210b, and 210c includes a resolution conversion unit 201a, b, c, a codec processing unit 202a, b, c, and a FIFO 203a, b, c. In each of the image processing units 210a, b, and c, different processing parameters are given for each unit according to a user's request, and processing according to the parameters is performed independently, but each codec is provided. Only specific encoding methods are supported.

  In the first example of this conventional video distribution system, the number of image processing units included in the image processing apparatus 200 is 3, and when the number of types of image processing requested by the user exceeds that number, it corresponds to that. I can't. In general, when a plurality of image processing units are provided to satisfy a plurality of image processing requests as in a conventional video distribution system, the contents that can be serviced are limited by the number of processing units. If the number of processing units is increased, the limit will be relaxed, but network camera servers and other devices that have the shooting function of the video distribution system are generally required to be downsized. In practice, the number of processing units is limited because the amount is limited.

  In order to remove the restriction on the number of processing units, a system having an image processing unit capable of changing image processing in a programmable manner is required.

  As such a conventional video distribution system, the image processing apparatus reads various conversion algorithms stored in the ROM in advance as needed, and responds to various requests of the user by causing the DSP to process the algorithms. There is something to be tried (for example, see Patent Document 1).

Japanese Patent Laid-Open No. 6-125411 (page 8, FIG. 1 and page 10, FIG. 4)

  On the other hand, when taking a video in real time and using the video for storage and distribution for monitoring purposes, it is not allowed to miss the video coming from the imaging device. For example, when video input is obtained from an NTSC-compliant video camera input that captures 30 frames of video per second, the time allowed for processing of one frame is only 1/30 seconds in principle, and that was mentioned earlier. The system is required to meet the demands of a wide variety of users.

  In particular, a network camera server capable of remotely operating the video shooting unit is required to have real time and low delay.

  However, in the technique described in Patent Document 1, a plurality of image processes are simply processed in parallel in a time division manner according to an algorithm, and there is no guarantee that the processes are completed in real time. For example, when multiple processes are performed at the same time, it is not possible to detect changes in the processing time required for processing by the image processing unit due to changes in the input image signal, etc. Could not be guaranteed. Further, when a service request is newly generated from a user, it has not been possible to determine whether or not all requests can be satisfied from the aspect of guaranteeing real-time performance.

  SUMMARY An advantage of some aspects of the invention is to provide a video distribution service while maintaining real-time performance, for example.

The above-described problems are solved by the image processing apparatus and the control method thereof according to the present invention. The image processing apparatus according to an aspect of the present invention, an image processing means for performing a plurality of compression processing against the image data, a distribution means for distributing the image data compressed by the image processing unit, said image processing measuring means means for measuring the processing power required for the single compression processing of a plurality of compression processing performed on the image data, based on a result of measurement by said measuring means, other than the compression process performed the measurement Prediction means for predicting the processing capacity required for the plurality of compression processes, and the image processing means has a predetermined total time required to complete each process based on a prediction result by the prediction means. characterized by executing the compression processing falls within a unit of time.

The image processing apparatus according to another aspect of the present invention, a distribution means for distributing an image processing means for performing a plurality of compression processing against the image data, the image data compressed by said image processing means, the image Measurement means for measuring the processing capacity required for the compression processing performed on the image data based on the first parameter by the processing means, and a new value performed on the image data used for the measurement based on the second parameter If accepting the compression process performed on the basis of the first parameter to the image data used in the measurement when receiving a request for compression processing, based on the results of measurement by said measuring means, said measuring prediction means for predicting a processing capability required for new compression, the results of prediction by the previous SL predicting means receiving the relative image data using performed based on the second parameter Based on the processing capacity required for it is compression processing with, accept the request for a new compression when it is judged that the completed within the unit of Jo Tokoro time, it is determined that not completed within the predetermined unit time And control means for rejecting a request for a new compression process .

Yet another aspect of the present invention relates to a method of controlling an image processing apparatus that performs image processing, an image processing step of performing a plurality of compression processing on the image data, the image data compressed by the image processing step a distribution step of distributing the step of measuring the processing power required for the single compression processing of a plurality of compression processing to be performed on image data by the image processing step, based on the result of measurement in said measuring step A prediction step for predicting the processing capability required for a plurality of compression processes other than the compression process for which the measurement has been performed , and the image processing step completes each process based on a prediction result in the prediction step total time to is characterized that you perform the compression process within a predetermined unit time.

Still another aspect of the present invention relates to an image processing apparatus control method, an image processing step for performing a plurality of compression processes on image data, and a distribution step for distributing the image data compressed in the image processing step. If, on the basis of a measurement step of measuring the processing power required for the compression process performed on the basis of the first parameter to the image data in the image processing step, the second parameter to the image data used for the measurement In the case where the compression processing performed based on the first parameter is received for the image data used for the measurement when a request for a new compression processing performed is received, based on the measurement result in the measurement step , prediction step you predict capacity required for the new compression processing performed based on the second parameter to the image data used for the measurement , On the basis of the processing capacity result and required for the accepted and compression of the prediction in the prediction step, if it is determined to be completed within a predetermined unit time accepted the request for a new compression of the predetermined A control step of rejecting a request for a new compression process when it is determined that the completion cannot be completed within a unit time .

  According to the present invention, in a video distribution system such as a network camera server used in a monitoring system with an emphasis on real-time characteristics, processing for an image processing apparatus requested by a user is completed within a predetermined unit time. In the video distribution system, when multiple requests from multiple users with various resolutions, compression methods, and quality overlap, which processing is processed in real time You will be able to judge whether you can finish As a result, it is possible to provide a video distribution service to a plurality of users without losing real-time characteristics.

  Furthermore, since the service is realized without mounting a plurality of processing units in parallel, the contents of the service (type of video compression method, resolution, Q value, frame rate) are processed in advance within a range not losing the real-time property. It becomes possible to provide services without being limited by the number.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a configuration diagram illustrating an image processing apparatus according to the present embodiment. In FIG. 1, the image processing apparatus 10 includes two components roughly divided into a processing control unit 100 and an image processing unit 200.

  First, the parameter value stored in the parameter register 104 in the processing control unit 100 is given to the image processing unit 200 as the parameter setting signal 152 by the processing order control unit 101. Next, a processing start signal 154 is given to the image processing unit 200 by the processing order control unit 101. If the process instructed here is an image compression process, the input image data 150 is processed based on the set parameter value, and the encoded data 151 is output. On the other hand, if the instructed process is an image expansion process, the input encoded data 151 is expanded based on the set parameters and output as image data 150. During the period when these processes are performed, the image processing unit 200 asserts the processing busy signal 153. The processing capacity measuring unit 102 measures the time during which the processing busy signal 153 is asserted.

  The processing capacity measurement unit 102 provides the measurement result to the processing capacity prediction unit 103. The processing capability prediction unit 103 predicts how much processing time is required for the subsequent image processing that is requested for processing from the parameters when the processing is performed and the processing capability measurement result, and performs processing order control. Presented in part 101. Upon receiving the presentation, the processing order control unit 101 determines which processing is to be performed next from the priority order of the requested processing and the processing capability of the image processing unit 200 required for the processing, and The processing unit 200 is instructed to perform the next process.

  FIG. 2 shows a detailed configuration of the image processing unit 200.

  2, the image processing unit 200 includes a resolution conversion unit 201, a codec processing unit 202, a FIFO 203, and a buffer 204. The input image data 150 is temporarily stored in the buffer 204 in units of frames, for example. The portion before the resolution conversion unit 201 handles image processing based on given parameters in one video input data unit (for example, one frame unit). If the instructed process is an image compression process, the resolution conversion unit 201 first reads the input image data 150 from the buffer 204, and based on the parameter setting signal 152 given from the parameter register 104, Resolution conversion processing is performed as necessary. The input of the image data 150 to the buffer 204 is governed by the speed of the input from the outside of the image processing unit 200, whereas the bit width and transfer speed in the transfer of the image data from the buffer to the resolution conversion unit 201 and thereafter. Can be made larger than the input speed of the image data 150. Therefore, during the time required for the image processing unit 200 to input one frame of the image data 150, the resolution conversion unit 201, the codec processing unit 202, and the FIFO 203 are used up to the limit of processing capability, and the same input is performed. Image compression processing based on a plurality of different parameters can be performed using a data frame image.

  The resolution conversion unit 201 passes the data after resolution conversion to the codec processing unit 202 in units corresponding to the encoding format specified by the parameters. The codec processing unit 202 compresses the input data while using the image processing memory 12 as necessary, and outputs the created compressed data to the FIFO 203. The FIFO 203 sends the data as encoded data 151 to the outside of the image processing unit 200 in response to an event such as accumulation of a predetermined amount or more of compressed data.

  Note that the resolution-converted image data is temporarily stored in the image processing memory 12 in addition to being sent to the codec processing unit 202. This is for the case where the same resolution conversion processing is required for the same input image data in the subsequent processing. If the same resolution processing as that previously performed in the subsequent processing is requested, the resolution conversion unit 201 reads the result of the resolution conversion processing previously performed from the image processing memory 12 and uses it as the codec. The data is supplied to the processing unit 202.

  On the other hand, if the instructed process is an image expansion process, the data flow direction is reversed. The input encoded data 151 is first stored in the FIFO 203. The codec processing unit 202 determines the encoding format of the input encoded data based on the parameter setting signal 152 given from the parameter register 104, sequentially reads the data from the FIFO 203, and performs the expansion processing. The decompressed image data is converted into the resolution required for display by the resolution conversion unit 201 as necessary, and stored in the buffer 204. The image data 150 stored in the buffer 204 is output to the outside of the image processing unit 200 as an image having a data rate suitable for outputting an image.

  Here, the buffer 204 is described as existing inside the image processing unit, but in some cases, the same memory as the image processing memory 12 can be used. Although not shown, in that case, the input image data 150 is temporarily stored directly in the image processing memory 12 without going through the resolution conversion unit 201, and the resolution conversion unit 201 stores the stored data in the image memory 12. The processing is performed while reading from.

  Note that the codec processing unit 202 of the image processing unit 200 asserts the processing busy signal 153 during the period during which the compression or expansion processing is performed.

  FIG. 3 shows a configuration example of a video distribution server incorporating the image processing apparatus 10 shown in FIG. 1 and a monitoring system configured using the video distribution server.

  Reference numeral 1 denotes a video distribution server, which captures and compresses a video signal input from the camera 6 or a camera 18 composed of a lens attached to the video distribution server 1 and a CCD, etc. Send to 5 etc. on the network. The transmitted video data is received by a client such as the mobile terminal 3 connected to the public line network 4 or the terminal 2 (for example, 2a to 2c) connected to the LAN line network 5, and is displayed by the respective viewer 31 or 21. The In some cases, the video data transmitted from the counterpart terminal is received, decompressed, and displayed on the connected monitor 7.

  Hereinafter, the internal configuration and basic operation of the video distribution server 1 will be described in detail.

  First, a video signal is taken in by the input image conversion device 11 from the camera 6 or a camera 18 constituted by a lens attached to the video distribution server 1 and a CCD. At this time, it is assumed that the data given from the camera 6 is an analog video signal based on a system such as NTSC / PAL / SECAM. Such an analog video signal is A / D converted by the input image converter 11 and then converted into an appropriate digital data format, for example, Y: U: V = 4: 2: 2, as an input to the image processor 10. The Further, although the video data output from the video input unit 18 is already in the digital format, the signal line from which the video data is output from the camera 18 conforms to the LVDS (Low Voltage Differential Signaling) format. The image is once received by the input image conversion device 11 and converted into a parallel data format that is easy for the image processing device 10 to handle. Then, it is input as image data 150 to the image processing apparatus 10.

  The image processing apparatus 10 that has acquired the image data 150 once holds the image data in the buffer 204, and then compresses the video based on the method instructed by the user. For example, compression is applied based on the M-JPEG system or the MPEG4 system. The compressed encoded data 151 is stored in the arithmetic RAM 15 connected to the microprocessor 13 via the microprocessor 13.

  When the encoded data is transferred to the operation RAM 15, the microprocessor 13 takes out the encoded data from the operation RAM 15 according to the program stored in the Flash ROM 14, and divides it into an appropriate size for delivery to the network. And processing such as adding a header. Then, via a communication PC card 171 (modem card, ISDN card, etc.) inserted in the PC card I / F 17 to a portable terminal on the public line network 4 or on the LAN line network 5 via the LAN interface 16 The encoded data is transmitted to other terminals.

  Further, the microprocessor 13 can not only distribute the encoded data on the network but also store it. For example, if a storage PC card 172 (such as a hard disk card or a Flash ROM card) is inserted into the PC card slot 17, video data can be recorded on the storage PC card 172. Further, the flash ROM 14 can be selected as the storage destination, and the arithmetic RAM 15 can be selected as the sustainable storage.

  Note that the video distribution server 1 obtains the DC power from the commercial AC power supply 91 via the AC-DC adapter 90 in the power supply unit 9 and further stabilizes the voltage to a voltage suitable for the operation of the IC in the apparatus. Power is supplied to the entire system after conversion by the power supply unit 9 each time.

  First, the operation when the video distribution server 1 distributes video to a plurality of users while predicting the performance of the image processing apparatus 10 will be described using FIGS. 4 to 6 with reference to FIGS. To do.

  FIG. 4 is a diagram showing an example of a parameter list of processing currently requested by the video distribution server 1. As shown in the figure, the parameter list describes parameters relating to the processing number, priority, processing type, encoding method, resolution, quality 1 and 2, and the number of connections. Here, it is assumed that the smaller the value indicating the priority, the higher the priority. Further, it is assumed that the contents of the parameter list are encoded and contained in the parameter register 104 of the image processing apparatus 10. In the present embodiment, as shown in the parameter list of FIG. 4, it is assumed that six processes with process numbers 0 to 5 are requested, and that each process number and the priority coincide.

  In FIG. 4, there is a process with priority 0 (process number 0). This process is treated as a reference process, and this image is first processed prior to all processes regardless of whether there is a processing request from the user. After that, the processing amount of the image processing unit required for performing the reference processing is measured, and the subsequent processing 1 to 5 is examined so as to enable processing in real time based on the measured amount. Become.

  FIG. 5 is a flowchart showing processing in the image processing apparatus 10.

  It is assumed that image data is continuously input from the camera 18 to the video distribution server 1 in units of frames, the frame update interval is 1/30 seconds, and the resolution of the input image is SVGA (800 × 600). First, an image in units of frames is input from the input image conversion device 11 to the image processing device 10. When the frame image data for one frame is stored in the buffer 204, the start of a new unit time synchronized with the timing is indicated (step S501).

  In the present embodiment, there is a process 0 with a priority 0 as described above. Since this processing is determined to be performed prior to all image processing, parameter information related to processing 0 such as [Motion-JPEG compression, resolution 320 × 240, quality Q = 50] is previously stored in the parameter register 104. The setting signal 152 is given to the resolution conversion unit 201 and the codec processing unit 202 (step S502).

  Thereby, first, when an instruction to start processing is given by the processing start signal 154, the resolution conversion unit 201 performs resolution conversion processing from 800 × 600 to 320 × 240 on the image data stored in the buffer 204, and resolution The converted image is sent to the codec processing unit 202. In this case, since Motion-JPEG processing is performed, the data is sequentially input to the codec processing unit 202 in units of 8 × 8 bit blocks. When the codec processing unit 202 receives the block from the data resolution conversion unit 201 and starts processing, the codec processing unit 202 asserts a processing busy signal 153. Then, Motion-JPEG encoding is performed using the given quality parameter Q = 50, and the processed encoded data is input to the FIFO 203. The encoded data input to the FIFO 203 is transferred to the arithmetic RAM 15 (step S503).

  When the processing of the image data corresponding to all 320 × 240 screens scheduled for the processing number 0 is completed, the processing busy signal 153 is negated. In response to this, the processing capacity measurement unit 102 measures the time during which the processing busy signal 153 has been asserted, and calculates the time required for the image processing of the processing number 0 (step S504).

  Assume that the time required for the image processing of process number 0 is t0. In response to the measurement result, the prediction performance calculation unit 132 extracts the contents of processes 1 to 5 requested subsequently from the parameter register 104 and predicts the processing time required for them (step S505).

  For example, it is assumed that the encoding process of the reference process with priority 0 is Motion-JPEG compression. When the resolution of the process is R0 and the time required for the process is t0, the processing time tn of the process n having the resolution Rn can be obtained as follows.

tn = (Rn / R0) × t0 × M × Q (1-1)
Where M is a factor factor in an empirically required encoding method,
M = 1 (when the encoding method is Motion-JPEG),
M = 1.05 (when the encoding method is MPEG4 and the processing frame is an I frame),
M = 1.75 (when the encoding method is MPEG4 and the processing frame is P frame),
Q: Quality factor (Here, it is set to 1 in all cases for simplification)

  When the above equation is calculated for n = 1 to 5, the processing time for each of the processes 1 to 5 is predicted. Next, it is determined whether or not all the processing times predicted in this way can be processed within the unit time (step S506). In this case, the determination in step S506 is performed by the following equation, for example.

(T0 + P) + (t1 + P) +... + (Tn + P) <processing unit time (1-2)
However, P: time required for parameter setting to the image processing unit,
n: Process number of the requested process

  In this embodiment, since the frame update interval is 1/30 seconds, the processing unit time is 1/30 seconds, and it is necessary to finish all the processing within that time. If n is increased in order and Formula 1-2 is satisfied even if it is increased to the point where the current request is accepted, the condition of step S506 is satisfied and the process proceeds to the YES branch, but the number of n is increased to the end. If this equation does not hold before this time, the process proceeds to the NO branch.

  FIG. 6 shows how this process is predicted. In the process of the first frame time in FIG. 6, it is predicted that the process up to the process 4 is processed within the process unit time as a result of the process performance prediction (that is, the expression 1-2 is established). When the process up to 5 is performed, it is determined that all processes are not completed within a predetermined time (step S506 → NO).

  In such a case, the processing priority is determined based on the policy (step S507). If the service policy of the video distribution server here is a priority priority policy, and processing that is not in time for real-time processing is not performed, the fifth processing with the lowest processing priority is the first. A decision is made not to implement in one frame time. Then, in response to the result, the remaining processes excluding the process 5 are sequentially performed (steps S508 to S510). When the process 4 is completed, the image processing unit 200 enters a standby state until the next new unit time starts (step S511).

  In the process of the second frame time in FIG. 6, since the compression process of process 3 is an MPEG4 I frame process, the process time is shorter than the first frame time, and all the processes up to process 5 are performed. It is determined that it can be performed (S506 → YES), and all requested processes are performed.

  In FIG. 4, process 2 is an M-JPEG decompression process. The flow of the process when the process is performed will be described with reference to FIGS. 2 and 3 again.

  In FIG. 3, a video is taken by a camera 22 attached to the terminal 2C, compressed by the Motion-JPEG method, and inputted to the video distribution system 1 from the terminal 2C via the LAN network 5. And The video distribution system 1 receives video data encoded in the Motion-JPEG format via the LAN I / F 16 and stores it in the arithmetic RAM 15. After removing the header for network communication, the microprocessor 13 prepares the image processing apparatus 10 to accept the encoded data as input, and then sends the encoded data to the FIFO 203 of the image processing unit 200. The contents of requested processing are transmitted to the register 104. When the predetermined order of processing comes, the image processing unit 200 expands the encoded data 151 using the codec processing unit 202 for expansion processing and outputs it to the resolution conversion unit 201. In this embodiment, since the resolution is output without being converted as it is, the decompressed image data 150 is transferred to the output image conversion device 19 via the buffer 204, where it is D / A converted and then externally connected. It is displayed on the monitor 7.

  A series of processing is performed as described above.

  In this embodiment, each component of the processing control unit 100 in the image processing apparatus 10 is described as an independent control unit. However, a dedicated controller or the like is prepared as the processing control unit 100 and realized by software. It is also possible to take a method. Furthermore, it is also possible to implement all processing of the processing control unit in the form of using dedicated software on the microprocessor 13 without preparing a dedicated controller as the processing control unit 100.

(Second Embodiment)
In the first embodiment, the calculation of the prediction performance in the processing capacity prediction unit is performed by a fixed prediction equation as shown in equations 1-1 and 1-2. However, in reality, there may be a discrepancy between the prediction result and the actual processing time depending on the content of the video to be shot.

  Therefore, in the present embodiment, FIGS. 1 to 5 are used as an example in which a difference between a prediction result and an actual processing time due to a change in a shooting target is detected and a feedback system is inserted so as to reduce the difference. This will be described with reference to FIGS. 1 to 5 are the same as those in the first embodiment, and a description thereof will be omitted here.

  FIG. 7 is a configuration example of the processing capability prediction unit 103 in the present embodiment. The processing capacity prediction unit 103 includes a prediction result comparison unit 131, a prediction performance calculation unit 132, and a prediction condition variable storage unit 133.

  8A and 8B are flowcharts showing processing in the image processing apparatus 10 according to the present embodiment.

  In the part indicated by “A” in this flowchart, the same process as the steps up to the part indicated by “A” in FIG. 5 (ie, steps S501 to S507) is performed at the beginning of this process. Show. Therefore, the description of these processes is omitted here, and the description starts with the processes after step S801.

  Since the contents of the process to be processed next are determined up to “A” in FIG. 5 (that is, steps S501 to S507), parameters are set from the parameter register 104 to the image processing unit 200 according to the contents (step S801). . When the parameter setting is completed, the processing order control unit 101 instructs the image processing unit 200 to perform the next processing using the processing start signal 154.

  Upon receiving the instruction, the image processing unit 200 starts image processing, and asserts a processing busy signal 153 while performing the processing (step S802). The processing capacity measurement unit 102 measures the time during which the processing busy signal 153 is asserted (step S803). Then, the measured processing time is compared with the processing time previously predicted by the prediction performance calculation unit 132 in step S505 in the prediction result comparison unit 131 (step S804).

  If the process has been completed within the predicted time (step S804 → YES), the process continues. That is, it is confirmed whether all scheduled image processing has been completed (step S807). If there is an unexecuted process (step S807 → NO), the next process is performed to execute the next process. Returning to the image processing parameter setting (step S801). If all the scheduled processes have been performed, a standby state is entered until the next new unit time starts (step S808).

  On the other hand, if the process has not been completed within the predicted time (step S804 → NO), first, it is examined whether or not the subsequent requested process can be processed by the start of the next unit time. This is performed (step S805). If the process is possible (step S805 → YES), the process returns to the previously scheduled process and the process is performed (step S807). On the other hand, when the subsequent requested processing cannot be performed by the start of the next unit time (step S805 → NO), the determination of the processing priority based on the policy is performed again (step S805). Step S806). This is to prevent a low priority process from being processed for a long time.

  The processing flow of the image processing unit 200 is generally as described above. However, the processing capability prediction unit 103 needs to simultaneously check the prediction condition variable. For this reason, the process branches from step S803 to step S809 in FIG. 8B, and the processing time measurement process 102 receives the result of measuring the assertion time of the processing busy signal 153, and the measured processing time is predicted performance. It is determined whether or not the calculation unit 132 matches the previously calculated result within a certain error range (step S809). If the measured value and the predicted value match within a certain error range (step S809 → YES), the process ends without correcting the prediction condition variable (S811). On the other hand, if the measured value and the predicted value are different (step S809 → NO), the prediction condition variable is modified so as to reduce the difference. For example, the following can be considered as a method of adding correction.

  First, the processing time prediction formula in the first embodiment is changed as follows. Assume that the encoding process of the reference process with priority 0 is Motion-JPEG. When the resolution of the process is R0 and the time required for the process is t0, the process time tn of the process n having the resolution Rn is as follows.

tn = (Rn / R0) * r * t0 * M * m * Q * C (2-1)
Where M is the factor in the encoding method,
M = 1 (when the encoding method is JPEG),
M = 1.05 + i (when the encoding method is MPEG4 and the processing frame is I frame),
M = 1.75 + p (when the encoding method is MPEG4 and the processing frame is P frame),
r: processing time correction term for resolution ratio,
i: correction term of processing time at the time of MPEG4, I frame processing,
p: correction term of processing time at the time of MPEG4, P frame processing,
m: correction term of processing time related to motion vector search range limitation during MPEG4, P frame processing,
Q: Quality factor (correction term due to differences in Motion-JPEG Q value, etc.),
C: Correction term based on the difference between compression processing and decompression processing

  Formula 1-1 of the first embodiment is i = p = 0 and r = M = m = Q = C = 1 in Formula 2-1, and it can be considered that this was fixed.

  Of the correction terms, m can increase the processing speed by limiting the search range of the motion vector when it is predicted that there is a motion in the image to be photographed by an instruction such as camera swinging. This is a correction term for taking that element into account.

  The condition variable for prediction is corrected by correcting the content of the correction term so that the difference between the predicted value and the actually measured value is small.

(Third embodiment)
The first and second embodiments described above relate to processing that currently accepts processing requests. However, in terms of distribution that maintains real-time video, how to service requests for new processing requests It is also necessary to consider from the perspective of whether or not

  In addition, as in the first embodiment, performing a specific definite process as a standard process regardless of whether or not a process request is received from a user with a priority of 0 may lead to effective use of resources in some cases. There may be no.

  On the other hand, in this embodiment, with respect to how to respond to a new service request, an example in which the processing capacity prediction process is used as a determination criterion, and an example in which the reference process is changed by the accepted new process. It shows using FIG. 9, FIG. 10, FIG. Since other figures are common to the first and second embodiments, description thereof will be omitted. However, it is assumed here that the resolution of the input image input from the camera 18 is XGA size (1024 × 768).

  FIG. 9A is a parameter list of processing for which the video distribution server 1 is requested at a certain time 1. At time 1, three processes from process 0 to process 2 are performed on the image processing apparatus 10. Here, a process when a request for a new process 3 is generated from the user will be described with reference to the flowchart of FIG.

  In FIG. 10, when a request for a new processing service is generated (step S1001), first, the image processing apparatus 10 measures and / or predicts the processing capability required for the entire currently accepted image processing (step S1002). This may be the actual measurement result of the reference process (process 0 at time 1) and the prediction result of other processes derived therefrom, and if a new service request can be waited for, the currently accepted image process is accepted. You may use the result measured about all.

  On the other hand, the processing capability prediction unit 103 estimates the processing capability of the image processing unit 10 required for the newly requested processing from the parameters of the processing that is newly requested for processing and the actual measurement result of the reference processing (step S1003).

  Then, it is estimated whether all the processes can be processed within the unit time even if the newly requested processes are accepted (step S1004). In the case of this embodiment, the processing time is relatively large, the resolution of the requested new processing is small, and the processing capability of the image processing unit is not significantly affected ( Step S1004 → YES), the request for a new service is accepted (Step S1005).

  When a new service request is accepted, if there is no absolute reference process with priority 0, it is determined whether or not the newly accepted process (in this case, process 3) is suitable as a reference process (step S1006). The reference process at time 1 was selected as process 0, and priority 1 higher than the other processes 1 and 2 was given. Comparing process 0 and process 3, it can be seen that the same Motion-JPEG process is used, but the resolution of process 3 is smaller than that of process 0. Here, when there is a policy that a smaller resolution is preferable as a reference policy as a reference process selection policy of the video distribution system 1, it is determined that the process 3 should be adopted as the reference process. (Step S1006 → YES). In that case, the reference process is changed (step S1007), and the priority of the process is also changed. Since the reference has changed, the prediction condition variable is also corrected at the same time (step S1008). Then, the correspondence to a series of new processing services is completed (S1009).

  Next, let us consider a case where three new processing requests are given to the video distribution server 1 at time 2 after a while has elapsed since time 1. FIG. 9B is a parameter list of processing for which the video distribution system 1 is requested at time 2. At time 2, four processes from process 0 to process 3 are performed on the image processing apparatus 10. Compared to FIG. 9A, since the reference process is changed from process 0 to process 3, the priority of process 3 is changed to 1, and the priority of other processes is changed to 2.

  Here, processing when a request for new processing 4 to 6 is generated from the user will be described with reference to FIGS. 10 and 11.

  FIG. 11 shows a prediction result of the time required for processing of a newly requested process through steps S1001 to S1003 in FIG. As can be seen from FIG. 9B and FIG. 11, among the newly requested processing, the processing 4 is performed, so that processing within the unit time becomes impossible (step S1004 → NO). The request for the new processing service is rejected (step S1010). On the other hand, with respect to the process 5, since it is determined from FIG. 11 that the process within the unit time is possible even if the process is added and implemented (step S1004 → YES), a request as a new process service is made. Accepted. However, since the resolution of the process 4 is larger than that of the process 3 that is the reference process at the time point 2, the change of the reference process does not occur (step S1006 → NO).

  Regarding process 6, the compression method, resolution, and quality are the same as process 0, and only the required frame rate is different. In this case, the code that is the output result of the image compression process performed in process 0 is It can be seen that the request is satisfied by copying the digitized data on the arithmetic RAM 15 and outputting it to the network. For this reason, a service can be added without newly setting a process in the image processing apparatus. Therefore, this process 6 is also added as a new process service. However, the process 6 is merged with the process 0 on the parameter list, so that the number of connections of the process 0 is 10.5, and the entry of the process 6 is deleted.

(Fourth embodiment)
In the first to third embodiments, the image compression process used for actual distribution is used as the reference process when performing performance prediction. However, various methods can be considered for selecting the reference process. .

  For example, when the video distribution server 1 is also accumulating images for monitoring, the processing can be adopted as the reference processing. When performing video accumulation for monitoring, it is necessary to finish the processing within real time, in particular to eliminate video omissions, and the load on the image processing device is limited by the prediction of the processing time in this proposal. Is very effective. An example of such a case is shown in FIG.

  FIG. 12 is a parameter list of requested processing in this embodiment. In this example, processing 1 is the highest priority processing because it is video storage for monitoring. Therefore, the priority is treated as 0 even though the number of connections is 0. In such a case, handling the process 1 as a reference process for performance prediction effectively uses the processing performance of the system. The image processed by the process 1 is stored on the storage PC card 172.

  In this example, the current process 1 is used only for image storage, but it goes without saying that this video can be used for distribution.

(Fifth embodiment)
Even when the processing performance of the image processing apparatus is variable, the processing performance prediction function is very effective. An example of such a case will be described with reference to FIGS.

  FIG. 13 shows an example of the configuration of the video distribution server 1 when the processing performance of the image processing apparatus is variable. 13 differs from FIG. 3 in that a battery 92 is connected to the power supply unit, a processing speed control unit 93 is connected to the power supply unit 9, and the processing speed control unit 93 connects to the image processing apparatus 10. On the other hand, a signal line for controlling the processing speed of the image processing unit is connected.

  In FIG. 13, the power supply unit 9 monitors how the power supply related to the video distribution server 1 is realized. When the power is mainly supplied from the battery 92, the power is supplied to the image processing apparatus 10 using the processing speed control unit 93 in order to reduce the power consumption of the video distribution system 1 in order to suppress the consumption of the battery 92. Control such as reducing the frequency of the clock that is running is performed.

  When the frequency of the clock supplied by the processing speed control unit 93 is lowered, the image processing apparatus 10 has a reduced image processing capability, and performs all the processes that could be performed before the frequency was lowered. Becomes difficult. FIG. 14 shows a parameter list of processing considering such a case. Here, a flag is set as to whether or not the quality 3 parameter is a process that requires real-time consideration. In the present embodiment, parameters for requesting real time are set for the processes 0 and 1. As processing priorities, processing 0 and processing 2 are the same. However, when the processing performance of the image processing apparatus 10 is degraded due to the presence of this parameter, processing 0 and processing 1 have priority. Will be distributed processing power. Specifically, first, the processing of the processing 1 as the reference processing is performed, and the processing time from the processing 1 to the processing 5 is predicted from the measurement result of the processing capacity, and the processing is performed within the range where the service can be performed in real time. The processing capability of the image processing apparatus 10 is distributed in the order of 0, 2, 3, 4, and 5. (Whether processing 4 or 5 is prioritized cannot be determined from the information in this parameter list alone, and other priority policies are used.)

(Sixth embodiment)
In the first to fifth embodiments, the processing busy signal 153 indicating that the processing of the codec processing unit 202 is mainly performed is used as a criterion for determining the processing performance. However, what determines the processing performance of the image processing unit 200 is the codec. It is not limited to the usage status of the processing unit 202. An example of such a case is shown in FIG.

  FIG. 15 shows the configuration of the image processing unit 200 in this embodiment. As can be seen from comparison with FIG. 2, in FIG. 15, in addition to the processing busy signal 153 output from the codec processing unit 202, the FIFO busy signal 155 output from the FIFO 203 is input to the processing capability measurement unit 102.

  The FIFO busy signal 155 is a signal line for notifying the degree of congestion of the internal bus used when transferring the encoded data 151 output from the codec processing unit 202 to the arithmetic RAM 15 connected to the microprocessor 13. is there. In general, when a compression method such as Motion-JPEG is used, if the Q value that is a quality parameter is changed in order to adjust the quality of the compressed image, the amount of encoded data output from the codec processing unit 202 is also changed. It changes greatly with it. For this reason, the degree of bus congestion at the time of transferring encoded data also changes according to the amount of data transfer, but in some cases, data may not be transferred during one unit processing time. In such a case, even if the processing capability of the codec processing unit 202 is sufficient, the real-time property of processing is lost as a result. The FIFO busy signal 155 is a signal used to indicate the transfer processing capability between the FIFO 203 and the arithmetic RAM 15 from such a viewpoint. This signal indicates information such as whether or not the data transfer between the FIFO 203 and the arithmetic RAM 15 is performed smoothly, and how much data is transferred between the FIFO 203 and the arithmetic RAM. Notify against. The processing capability measuring unit 102 uses the processing information relating to the data transfer and the processing busy signal 153 indicating the usage status of the codec processing unit 202 as a whole as a whole processing capability of the image processing unit 200 required for processing. And the result is passed to the processing capability prediction unit 103 for use.

(Seventh embodiment)
In the above-described embodiment, all the image data 151 input from the input image conversion apparatus 11 to the image processing apparatus 10 to be subjected to compression processing is temporarily stored in the buffer 204 for one frame, from which data is read out at a high speed and processed. As a result, a plurality of image processes are realized within a unit time required to input an image for one frame.

  On the other hand, in the present embodiment, FIG. 16 is used as an example of realizing equivalent processing without accumulating an image for one frame in the buffer 204.

  In FIG. 16, unlike FIGS. 2 and 15, line buffers 205 a and 205 b are prepared instead of the buffer 204. These line buffers 205a and 205b have a double width of the maximum horizontal value of the image size inputted according to the format of Y: U: V = 4: 2: 2, for example, and can store data for 16 lines. Is. In the present embodiment, if XGA size data is input as image data, the size of the line buffer is 1024 × 2 = 2048, so that it is 2048 bytes × 16 lines.

  As the image data 150, video data is alternately input to the line buffers 205a and 205b every 16 lines. When the data for 16 lines is accumulated in the line buffer, the resolution conversion unit 201 reads the data at high speed, converts the data into block units corresponding to the compression processing, and supplies the data to the codec processing unit 202. At the same time, the resolution conversion processing is performed. To the required resolution for compression processing. The resolution processing conversion result is stored in the image processing memory 12.

In this case, as the standard process, it is desirable to use a process having a resolution as high as possible in order to minimize the load of reading data, from the viewpoint of optimizing the overall processing performance. When such processing is performed, processing performance prediction based on the reference processing is performed in units of this line buffer. Then, at the stage where the reference processing in units of line buffers is completed, the codec conversion unit 202 reads out the data with the necessary resolution from the image memory 12 according to the predicted result, and sequentially performs the next compression processing. Go. In this case, the result of the prediction performed in the head line buffer is also applied to processing in other line buffer units of the same frame. If an inconvenience occurs when processing the other line buffer in the same frame as a result of prediction in the first line buffer, the prediction result of the process is reviewed at the time when the inconvenience occurs. The result is applied to subsequent processing of the same frame.

(Other embodiments)
As mentioned above, although embodiment of this invention was explained in full detail, this invention may be applied to the system comprised from several apparatuses, and may be applied to the apparatus which consists of one apparatus.

  In the present invention, a software program that realizes the functions of the above-described embodiments is directly or remotely supplied to a system or apparatus, and the computer of the system or apparatus reads and executes the supplied program code. Is also achieved. In that case, as long as it has the function of a program, the form does not need to be a program.

  Therefore, in order to realize the functional processing of the present invention with a computer, the program code itself installed in the computer and the storage medium storing the program also constitute the present invention. In other words, the claims of the present invention include the computer program itself for realizing the functional processing of the present invention and a storage medium storing the program.

  In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.

  As a storage medium for supplying the program, for example, flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R).

  As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a storage medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the claims of the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of the processes.

  Furthermore, after the program read from the storage medium is written to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a figure which shows the structure of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the detailed structure of the image process part in the image processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the monitoring system using the video delivery system in the 1st Embodiment of this invention. It is a figure which shows an example of the parameter list | wrist of the process requested | required of the video delivery system in the 1st Embodiment of this invention. It is a flowchart which shows the process in the image processing apparatus in the 1st Embodiment of this invention. It is a figure explaining the processing performance prediction process in the 1st Embodiment of this invention. It is a figure which shows the structure of the processing capability estimation part in the 2nd Embodiment of this invention. , It is a flowchart which shows the process in the image processing apparatus in the 2nd Embodiment of this invention. , It is a figure which shows the example of the parameter list | wrist of the process requested | required of the video delivery system in the 3rd Embodiment of this invention. It is a flowchart which shows the flow of a process at the time of determining whether the acceptance of the new request process in the 3rd Embodiment of this invention is possible. It is a figure which shows the result of the processing performance prediction in the 3rd Embodiment of this invention. It is a figure which shows an example of the parameter list | wrist of the process requested | required of the video delivery system in the 4th Embodiment of this invention. It is a figure which shows the structure of the video delivery system in the 5th Embodiment of this invention. It is a figure which shows the example of the parameter list | wrist of the process requested | required of the video delivery system in the 5th Embodiment of this invention. It is a figure which shows the detailed structure of the image process part in the image processing apparatus which concerns on the 6th Embodiment of this invention. It is a figure which shows the detailed structure of the image process part in the image processing apparatus which concerns on the 7th Embodiment of this invention. It is a figure which shows the structure of the image processing apparatus in the conventional video delivery system.

Claims (11)

Image processing means for performing a plurality of compression processing against the image data,
Distribution means for distributing the image data compressed by the image processing means;
Measuring means said image processing means measures the capacity required for the single compression processing of a plurality of compression processing performed on the image data,
Predicting means for predicting the processing capacity required for a plurality of compression processes other than the compression process that performed the measurement , based on the measurement result by the measurement means;
Said image processing means on the basis of the results of prediction by the prediction unit, the image total time it takes to complete each process is characterized in that perform compression processing falls within a predetermined unit time processing apparatus. The measurement was in accordance with the by that meter Hakayui result to said measuring means processing power required for the plurality of compression processing other than the compression process performed, characterized by having a correction means for correcting the predicted condition of the prediction means The image processing apparatus according to claim 1. Image processing means for performing a plurality of compression processing against the image data,
Distribution means for distributing the image data compressed by the image processing means;
Measuring means for measuring the processing capacity required for the compression processing performed by the image processing means on the image data based on the first parameter ;
Compression processing performed on the image data used for the measurement based on the first parameter when a request for a new compression processing performed on the image data used for the measurement based on the second parameter is received If the are accepted, and predicting means based on the result of measurement by said measuring means, for predicting the processing power required for a new compression processing performed based on the second parameter to the image data used for the measurement ,
Based on the processing power required for the compression process is accepting the results of the prediction by the previous SL predicting means, receiving a request for a new compression process when it is judged that the completed within the unit of Jo Tokoro time, the predetermined A control means for rejecting a request for a new compression process when it is determined that it cannot be completed within a unit time ;
An image processing apparatus comprising: Accepts the request of a new compression processing based on a result the second parameter to the control hand stage, the resolution based on the second parameter, wherein when less than the resolution based on the first parameter, said measuring means Measures the processing capacity required for the new compression processing based on the second parameter for the image data used for the measurement,
The prediction means is a compression process other than the new compression process based on the processing capacity required for the new compression process based on the second parameter for the image data used for the measurement, measured by the measurement means. The image processing apparatus according to claim 3, wherein the processing capability required when receiving the request is predicted .   5. The image processing apparatus according to claim 1, wherein the predetermined unit time is a time required to input image data of one frame. The image processing apparatus according to claim 1 , wherein the image processing unit selects a compression process to be executed based on information related to a priority of the compression process. The predicting means predicts the processing capacity required for the decompression process on the image data based on the measurement result of the measuring means;
The image processing apparatus according to claim 1, wherein the image processing unit executes the expansion processing based on a result of prediction by the prediction unit. A method for controlling an image picture processor,
An image processing step for performing a plurality of compression processes on the image data;
A distribution step of distributing the image data compressed in the image processing step;
A step of measuring the processing power required for the single compression processing of a plurality of compression processing to be performed on image data by the image processing step,
On the basis of the results of the measurement in the measurement step, and a prediction step of predicting a processing capability required for the plurality of compression processing other than the compression process performing the measurement,
In the image processing step , based on the prediction result in the prediction step, a compression process is performed in which the total time taken to complete each process falls within a predetermined unit time.
Method of controlling an image processing apparatus according to claim and this. A method for controlling an image picture processor,
An image processing step for performing a plurality of compression processes on the image data;
A distribution step of distributing the image data compressed in the image processing step;
A measurement step for measuring the processing capacity required for the compression processing performed on the image data based on the first parameter in the image processing step;
Compression processing performed on the image data used for the measurement based on the first parameter when a request for a new compression processing performed on the image data used for the measurement based on the second parameter is received If the are accepted on the basis of the results of the measurement in the measurement step, I predict you predict capacity required for the new compression processing on the image data performed based on the second parameter used for the measurement Steps,
Based on the prediction result in the prediction step and the processing capacity required for the accepted compression process, if it is determined that the process can be completed within a predetermined unit time, a request for a new compression process is accepted, and the predetermined unit A control step for rejecting a request for a new compression process when it is determined that the process cannot be completed in time;
A control method for an image processing apparatus, comprising: On the computer,
An image processing procedure for performing a plurality of compression processes on the image data;
A delivery procedure for delivering the image data compressed in the image processing procedure;
A measurement procedure for measuring the processing capacity required for one of the plurality of compression processes performed on the image data in the image processing procedure;
Based on the measurement result in the measurement procedure, a prediction procedure for predicting the processing capacity required for a plurality of compression processes other than the compression process that performed the measurement is performed,
The image processing procedure executes a compression process in which the total time required to complete each process is within a predetermined unit time based on the prediction result of the prediction procedure.
A program characterized by that . On the computer,
An image processing procedure for performing a plurality of compression processes on the image data;
A delivery procedure for delivering the image data compressed in the image processing procedure;
A measurement procedure for measuring the processing capacity required for the compression processing performed on the image data based on the first parameter in the image processing procedure;
Compression processing performed on the image data used for the measurement based on the first parameter when a request for a new compression processing performed on the image data used for the measurement based on the second parameter is received A prediction procedure for predicting the processing capacity required for a new compression process based on the second parameter for the image data used for the measurement, based on the measurement result in the measurement procedure, ,
Based on the result of prediction by the prediction procedure and the processing capability required for the accepted compression process, if it is determined that the process can be completed within a predetermined unit time, a request for a new compression process is accepted, and the predetermined unit time A control procedure for rejecting a request for a new compression process when it is determined that the process cannot be completed,
A program characterized by having executed.

Images