JP2007323393A - Image processor and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To continue entire image processing, even when a storage resource is lacking, in image processing by executing a plurality of threads in parallel which correspond to image processing modules connected in pipe line configurations. <P>SOLUTION: Every time a thread corresponding to each module requests to acquire a memory, it is determined whether the requested memory area can be secured. When the memory area can be secured, it is assigned, or when the memory area cannot be secured, operation of a requesting thread is stopped, and acquisition request information is registered in a queue (according to execution priorities of threads). When memory release is requested from an arbitrary thread, the following processing are performed from the acquisition request information registered at the top of the queue: releasing the requested memory area (620), determining whether the memory area requested by the acquisition request information in the queue can be secured (624, 626), extracting the acquisition request information from the queue when the requested memory area can be secured, releasing the thread from the operation stop state, and assigning the memory area (628, 630). <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and a program, and in particular, an image processing apparatus including an image processing unit in which a plurality of image processing modules are connected in a pipeline form or a directed acyclic graph form, and a computer. The present invention relates to an image processing program for functioning as an image processing apparatus.

  In an image processing apparatus that performs image processing on input image data, a DTP (desktop publishing) system that can handle images, a printing system that records an image represented by input image data on a recording material, etc. Various kinds of image processing such as enlargement / reduction, rotation, affine transformation, color conversion, filter processing, and image composition are performed on the image data that has been processed. In these devices and systems, if the attributes of the input image data and the contents, procedures, parameters, etc. of the image processing for the image data are fixed, the image processing may be performed by specially designed hardware. However, for example, when various image data with different color space and the number of bits per pixel are input, or when the contents, procedures, and parameters of image processing are changed variously, the image processing to be executed is more flexible. A changeable configuration is required.

  In order to satisfy such a requirement, for example, in Patent Document 1, a programmable processing module is connected to a pipeline form or a DAG (Directed Acyclic Graph) form to perform desired image processing. Technologies that enable this have been proposed. The technology described in Patent Literature 1 is configured so that the contents of the arithmetic processing in each of the plurality of programmable arithmetic processing units and the connection form of each programmable arithmetic processing unit by the network unit can be freely set from the outside through the host control means. As a result, a digital video signal processing apparatus capable of high-speed and advanced arithmetic processing and having a high degree of freedom for function change and system change is realized.

  Further, the following techniques are known in relation to the above. In other words, Patent Document 2 receives input data from a single storage means for input among a plurality of storage means and an output unit, which is one of a plurality of storage means and time-division processing, and outputs it. A data processing means for storing the processing results in a single storage means for use, and an activation priority determining means for determining the execution priority of the execution unit from the information of the data storage amount of the storage means and determining the activation priority of the execution unit A processor with such is disclosed.

  Further, in Patent Document 3, in a multiprocessor system, a stall time (based on the priority of a single processing unit and the dependency of a processing result is stopped without performing processing until a condition that allows the next processing to be started is satisfied. A technique for performing scheduling so as not to occur) is disclosed.

Further, in Patent Document 4, a minimum amount of image memory required for a plurality of image processing processes operating in parallel is calculated in advance, and a necessary amount of image memory calculated at the time of starting the system is reserved and is not reserved. A technique is disclosed in which each image processing process is prevented from being interrupted and the efficiency of memory allocation is improved by notifying each image processing process when the memory usage of the memory changes dynamically.
JP-A-5-260373 JP 2004-287883 A Japanese Patent Laid-Open No. 10-232856 JP 2003-091425 A

  As in the technique described in Patent Document 1, an image processing unit is constructed by connecting a plurality of types of image processing modules to a pipeline form or a DAG form, and individual image processing modules constituting the image processing unit are arranged in parallel. When the desired image processing is performed by operating, each image processing module repeats the image processing in units of a certain amount of image data while repeatedly acquiring and releasing the memory necessary for executing the image processing. However, in this configuration, when a situation occurs in which a memory necessary for executing image processing cannot be acquired in a certain image processing module, image processing cannot be continued in the above image processing module. There was a problem that the entire image processing that was in progress also failed.

  The technique described in Patent Document 4 reserves the minimum amount of image memory required for each image processing process at the time of system startup, so that each image processing process operates with the minimum amount of image memory. During this time, it is possible to avoid the failure of the entire image processing due to the fact that a specific image processing process cannot acquire the memory. However, in the technique described in Patent Document 4, a necessary amount of memory is acquired in a situation where a certain image processing process requires a larger amount of memory than when a normal image is handled in order to handle a high-resolution image. If not, memory shortage error processing is performed. Therefore, even if the user desires to execute image processing on a high-resolution image, if the memory shortage occurs, the image processing desired by the user is not performed. It will be.

  The present invention has been made in consideration of the above facts, and a plurality of execution unit programs corresponding to different image processing modules among a plurality of image processing modules connected in a pipeline form or a directed acyclic graph form. The object of the present invention is to obtain an image processing apparatus and an image processing program capable of continuing the entire image processing even if storage resources are temporarily insufficient in a configuration in which image processing is performed in parallel.

  In order to achieve the above object, an image processing apparatus according to the first aspect of the present invention acquires image data from the previous stage of its own module, performs predetermined image processing on the acquired image data, and receives the image processing. Alternatively, a plurality of image processing modules having a function of using a storage resource to output the processing result of the image processing to the subsequent stage of the own module are connected in a pipeline form or a directed acyclic graph form. A plurality of execution unit programs corresponding to different or different image processing modules and requesting allocation and release of storage resources used by the corresponding image processing modules are executed in parallel by the program execution resources; Every time storage resource allocation is requested by the running image processor and any execution unit program It is determined whether or not the requested storage resource can be allocated to the allocation request source execution unit program. When it is determined that the allocation is not possible, the operation of the allocation request source execution unit program is stopped and the allocation is performed. When it is determined that the requested storage resource can be allocated to the execution unit program of the allocation request source that stopped the operation, the operation of the execution unit program of the allocation request source that stopped the operation is restarted. And management means for allocating the storage resource requested to be allocated to the execution unit program of the allocation request source that has resumed the operation.

  The image processing unit according to the first aspect of the present invention acquires image data from the previous stage of its own module, performs predetermined image processing on the acquired image data, and the image data subjected to the image processing or the processing result of the image processing A plurality of image processing modules having a function of using a storage resource (for example, memory, HDD (Hard Disk Drive), etc.) to output to the subsequent stage of the own module is in a pipeline form or a directed acyclic graph form Concatenated with. The image processing unit also includes a plurality of execution unit programs corresponding to different image processing modules or a plurality of execution unit programs that request allocation and release of storage resources used by the corresponding image processing modules. It works by being executed in parallel. As the program execution resource, for example, a CPU, an arithmetic unit for MMX (MultiMedia eXtention), an arithmetic unit for SSE (Streaming SIMD Extension), or a DSP (Digital Signal Processor) provided separately from the CPU is used. For example.

  Each of the plurality of execution unit programs can be executed in parallel by a program execution resource, for example, as an independent process or thread. In addition, when a single execution unit program corresponds to a plurality of image processing modules, a plurality of image processing modules corresponding to the same execution unit program are parallelized with program execution resources as threads that share storage resources, for example. Can be executed. Thus, according to the first aspect of the present invention, the individual image processing modules constituting the image processing unit operate in parallel, and from each execution unit program, an allocation request for storage resources used by the corresponding image processing module and Release request is output asynchronously.

  Here, when a certain execution unit program outputs a storage resource allocation request, if there is a shortage of allocatable storage resources, the corresponding image processing module falls into an inexecutable state. In the past, an error was output and the entire image processing in the image processing unit was stopped, but the amount of storage resources that can be allocated is often fluctuating constantly, and was requested at the timing when allocation was requested. Even if the storage resource is not assignable, the requested storage resource may change to an assignable state over time.

  Based on the above, the management means according to the first aspect of the invention allocates the storage resource requested for allocation to the execution unit program of the allocation request source every time the allocation of the storage resource is requested by an arbitrary execution unit program. It is determined whether or not allocation is possible, and when it is determined that allocation is not possible, the operation of the execution unit program of the allocation request source is stopped. In addition, when it is determined that the storage resource requested to be allocated can be allocated to the execution unit program of the allocation request source that stopped the operation, the operation of the execution unit program of the allocation request source that stopped the operation is resumed. The storage resource requested for allocation is allocated to the execution unit program of the allocation request source that has resumed operation.

  Thus, according to the first aspect of the present invention, when the storage resource requested to be allocated cannot be allocated, the operation of the execution unit program of the allocation request source is stopped without outputting an error. When the image processing by the image processing module corresponding to the execution unit program is interrupted and then the storage resource requested to be allocated is changed to an allocatable state, the operation of the execution unit program that has stopped the operation is resumed ( The image processing by the image processing module is resumed), and the storage resource requested to be allocated is allocated. Therefore, among the plurality of image processing modules connected in the pipeline form or the directed acyclic graph form, different image processing modules are used. In a configuration for executing image processing by executing a plurality of corresponding execution unit programs in parallel, a storage resource It is possible also to continue the entire image processing temporarily insufficient.

  In the first aspect of the present invention, the image processing unit includes a buffer composed of storage resources at least one of the front and rear stages of each of the plurality of image processing modules. When the image processing module is connected to the previous stage of the module, the image data output from the previous image processing module is written to the buffer, and when the image processing module is connected to the subsequent stage of the module, It is desirable that a buffer module for reading out the image data stored in the buffer by the image processing module in the subsequent stage is connected. In this case, the plurality of execution unit programs correspond to at least one of different single or plural image processing modules and different single or plural buffer modules, but the buffer module also uses the storage resource as a buffer. Therefore, the execution unit program corresponding to the buffer module is configured to request allocation and release of storage resources used by the corresponding buffer module.

  Each image processing module constituting the image processing unit has a unit (for example, a pixel unit, one line unit, a plurality of line units, a surface unit, etc.) that can be easily processed according to the type and content of the image processing to be executed. However, in order to be able to connect the image processing modules in an arbitrary order and process them in a coordinated manner, the output units of all the image processing modules are aligned, or each image processing module is set to an arbitrary input unit. It is necessary to configure so as to be compatible, and the configuration of each image processing module becomes complicated. On the other hand, as described above, if a buffer module is connected to at least one of the preceding stage and the succeeding stage of each of the plurality of image processing modules, acquisition and output of image data in each image processing module can be performed. This can be performed in units of size according to the type and contents, and the configuration of each image processing module can be simplified.

  Further, in the invention according to claim 1 or claim 2, when the operation of the execution unit program of the allocation request source is stopped by determining that the storage resource requested for allocation cannot be allocated, As described in claim 3, allocation is requested while there is an execution unit program of an allocation request source whose operation is stopped by determining that the storage resource requested for allocation cannot be allocated. It may be configured to determine periodically (for example, at a constant time period) whether or not the storage resource can be allocated to the execution unit program of the allocation request source. Thus, every time a storage resource is released in accordance with a request from another execution unit program, the storage resource requested to be allocated can be allocated to the execution unit program of the allocation request source. It may be configured to determine whether it is. As a result, it is possible to allocate the storage resource requested to be allocated while suppressing the load applied to the management means, as compared with the mode of constantly monitoring whether the storage resource requested to be allocated can be allocated. Can be detected early.

  Further, in consideration of the fact that there is a high possibility that the storage resource amount required by each execution unit program at the time of allocation request is different, in the invention according to any one of claims 1 to 4, each execution program The unit program is preferably configured so as to notify the storage resource amount required at the time of the storage resource allocation request, and the management means can allocate the storage resource amount (for example, claim In the invention described in item 4, the total storage resource amount that can be allocated and the total value of the storage resource amount that has been released in accordance with a request from another execution unit program becomes the above-mentioned allocationable storage resource amount). By determining whether or not the required storage resource amount is exceeded, it is determined whether or not the storage resource requested for allocation can be allocated to the execution unit program of the allocation request source. It is preferably configured to. As a result, it is possible to accurately determine whether or not the storage resource requested to be allocated can be allocated, and improve the use efficiency of the storage resource as compared with the case where the amount of storage resource allocated by one allocation request is fixed. be able to.

  Further, when the storage resource requested to be allocated cannot be allocated, a program other than the execution unit program of the allocation request source (for example, another execution unit program or a plurality of execution units corresponding to the image processing unit according to the present invention) Depending on the use of storage resources by other programs that are executed in parallel with the program (including the other programs), it is very difficult to allocate the requested storage resources. It can take a long time. In that case, since the processing in the image processing unit also takes a long time, there is a possibility that a large burden is placed on the user waiting for the completion of the processing in the image processing unit. For this reason, in the invention according to any one of claims 1 to 5, the management means operates by determining that the storage resource requested to be allocated cannot be allocated, as described in claim 6, for example. It is preferable that when the operation stop time of the execution unit program of the allocation request source that has stopped the operation exceeds a predetermined time, an error is output to stop the processing by the image processing unit. As a result, it is possible to prevent the user waiting for completion of the processing in the image processing unit from being subjected to a large burden.

  Further, in the invention according to any one of claims 1 to 5, the management means determines that the storage resource requested to be allocated from any execution unit program cannot be allocated, as described in claim 7. In addition, all other execution unit programs corresponding to the image processing unit stop the operation by determining that the storage resource requested to be allocated cannot be allocated, or correspondingly If the image processing module is in the second state in which the image data cannot be acquired from the previous stage of the module and the processing is stopped, an error may be output to stop the processing by the image processing unit. Good. As described above, among the execution unit programs corresponding to the image processing unit, all execution unit programs other than the execution request source program of the assignment request source are in the first state or the second state, and the assignment request source. If the requested storage resource cannot be allocated to the execution unit program, the processing in the image processing unit is completely stopped. In such a case, an error is output to stop the processing by the image processing unit. As a result, it is possible to prevent the user waiting for completion of the processing in the image processing unit from being subjected to a large burden.

  In addition, in the inventions according to claims 6 and 7, to stop the processing by the image processing unit, for example, the operation of the requesting execution unit program that has stopped the operation is restarted, and the execution unit in which the operation is restarted This can be realized by returning an error in response to a storage resource allocation request from the program.

  Also, in the invention according to any one of claims 1 to 7, the management means operates by determining that the storage resource requested to be allocated cannot be allocated, for example, as described in claim 8. The allocation request from the execution unit program of the allocation request source that has been stopped is registered in the queue, and the allocation request registered in the queue is extracted at an arbitrary timing, and the storage in which the allocation is requested by the extracted allocation request It can be configured to determine whether or not allocation to the execution unit program of the allocation request source corresponding to the allocation request from which the resource has been extracted is possible. As a result, even when there are multiple execution unit programs whose operation has been stopped by determining that the storage resource requested to be allocated cannot be allocated, whether these storage resources can be stored and the storage resource can be allocated. The process of determining whether or not at an arbitrary timing is simplified. Note that the above-described management by the queue is effective particularly when there are a plurality of execution unit programs whose operations have been stopped. Therefore, the management is performed only when there are a plurality of execution unit programs whose operations have been stopped. Management by a matrix may be performed.

  Further, in the invention described in claim 8, the management means can extract the allocation request from the queue and allocate the storage resource requested to be allocated by the extracted allocation request, as described in claim 9, for example. As a result of determining whether or not the storage resource is not allocatable as a result of determining whether or not the allocation request taken out is registered again in the queue and a plurality of allocation requests are registered in the queue In addition, a process of taking out a single allocation request from the queue and determining whether or not the storage resource requested for allocation by the single allocation request that has been taken out can be allocated is assigned to a plurality of allocation requests. It can be configured to be performed in the order of registration in the queue. As described above, it is determined that the storage resource requested to be allocated cannot be allocated by taking out the allocation request from the queue in the order of registration in the queue and determining whether the storage resource can be allocated. It is possible to average the operation stop times of a plurality of execution unit programs whose operations have been stopped.

  Further, in the invention according to claim 8, the management means takes out a single assignment request from the queue when a plurality of assignment requests are registered in the queue, for example as described in claim 10. The process of determining whether or not the storage resource requested for allocation can be allocated by the single allocation request that has been taken out, the execution unit of the allocation request source corresponding to each allocation request registered in the queue You may comprise so that it may carry out in descending order of the execution priority of a program. In this case, the operation stop time of the execution unit program having the higher execution priority is preferentially shortened among the plurality of execution unit programs whose operation is stopped by determining that the storage resource requested for allocation cannot be allocated. As a result, it is possible to improve the processing efficiency of the image processing unit when a shortage of storage resources occurs.

  Further, in the invention according to any one of claims 8 to 10, when the management unit requests the allocation of storage resources from the first execution unit program, for example, as described in claim 11, When an allocation request from the second execution unit program is already registered in the queue, the allocation request from the first execution unit program is registered in the queue as it is, or from the first execution unit program It is determined whether or not the storage resource requested to be allocated can be allocated by the allocation request, and when it is determined that the storage resource can be allocated, the storage resource requested to be allocated is allocated to the first execution unit program, and the storage resource Can be configured to register an allocation request from the first execution unit program in the queue. .

  Further, in the invention according to any one of claims 8 to 10, when the management unit requests the allocation of storage resources from the first execution unit program, for example, as described in claim 12, When an allocation request from the second execution unit program is already registered in the queue, it is determined whether or not the execution priority of the first execution unit program is higher than the execution priority of the second execution unit program When the execution priority of the first execution unit program is higher than the execution priority of the second execution unit program, the storage resource requested for allocation can be allocated by the allocation request from the first execution unit program. If the storage resource can be allocated, the storage resource requested to be allocated is allocated to the first execution unit program, and the storage resource must be allocated. While the allocation request from the first execution unit program is registered in the queue, when the execution priority of the first execution unit program is less than or equal to the execution priority of the second execution unit program, the first execution unit program The allocation request may be registered in the queue as it is.

  Further, in the invention according to claim 10 or claim 12, the management means according to the result of estimating in advance the processing load of each image processing module constituting the image processing unit, as described in claim 13, for example. The execution priority of each corresponding execution unit program can be set. The setting of the execution priority of each execution unit program according to the processing load of each image processing module increases the execution priority of the execution unit program corresponding to the image processing module estimated to have a high processing load, for example. Thus, it can be realized by lowering the execution priority of an execution unit program in which there is no image processing module estimated to have a high processing load among the corresponding image processing modules. As described above, the execution priority of each execution unit program is set appropriately by setting the execution priority of each corresponding execution unit program according to the result of estimating the processing load of each image processing module in advance. It can be set, and the efficiency of processing in the image processing unit when storage resources are insufficient can be realized.

  Further, in the invention according to claim 10 or claim 12, the management means, for example, as described in claim 14, is a pipeline form or a directed acyclic graph form of individual image processing modules constituting the image processing unit. The execution priority of the corresponding individual execution unit program is initially set according to the position in the connection form, and the execution priority of the individual execution unit program is changed according to the progress of the image processing in the image processing unit. You may comprise. The initial setting of the execution priority is such that, for example, the execution priority of the corresponding execution unit program becomes higher as the position of the image processing module in the connected form of the pipeline form or the directed acyclic graph form becomes the front side. The execution priority of the execution unit program can be changed according to the degree of progress of the image processing. For example, as the image processing proceeds, the execution priority of the execution unit program corresponding to the image processing module located on the preceding stage is changed. The execution priority can be lowered and the execution priority of the execution unit program corresponding to the image processing module located on the subsequent stage side can be increased. As a result, the execution priority of each execution unit program can be optimized regardless of changes in the degree of progress of processing in the image processing unit, and the processing in the image processing unit can be updated when a shortage of storage resources occurs. Can be realized.

  Further, in the invention described in claim 14, as in the invention described in claim 2, the image processing unit includes a buffer composed of storage resources in at least one of the front and rear stages of each of the plurality of image processing modules. When the image processing module is connected to the previous stage of the module, the image data output from the previous image processing module is written to the buffer, and the image processing module is connected to the subsequent stage of the module. A buffer module for reading image data stored in the buffer by a subsequent image processing module is connected, and the execution unit program is different from one another or a plurality of image processing modules and from one another. Corresponding to at least one of the buffer modules When the corresponding execution unit program is configured to request the allocation and release of the storage resources used by the corresponding buffer module, the management means, for example, as the degree of progress of the image processing, as described in claim 15 The ratio of the data amount of the image data stored in each buffer module to the unit data amount when the image processing module subsequent to the buffer module acquires image data from the individual buffer module is used. The execution priority of each corresponding execution unit program can be changed according to the ratio in the module.

  Further, in the invention described in claim 14, for example, as described in claim 16, the management means indicates the number of image data acquisition failures (from the preceding module) in each image processing module as the degree of progress of the image processing. It is also possible to change the execution priority of each corresponding execution unit program according to the number of acquisition failures in each image processing module.

  An image processing program according to claim 17 is a computer having a storage resource and a program execution resource, acquires image data from the previous stage of its own module, performs predetermined image processing on the acquired image data, and executes the image processing A plurality of image processing modules having a function of using a storage resource to output image data that has undergone processing or a processing result of the image processing to a subsequent stage of the own module in a pipeline form or a directed acyclic graph form Multiple execution unit programs that are linked and correspond to single or multiple different image processing modules and that require allocation and release of storage resources used by the corresponding image processing modules are executed in parallel by the program execution resources Image processing unit that operates by being executed, and an arbitrary execution unit program Each time a storage resource allocation is requested from a program, it is determined whether or not the storage resource requested for allocation can be allocated to the execution unit program of the allocation request source. The operation of the original execution unit program is stopped, and the operation is stopped when it is determined that the storage resource requested to be allocated can be assigned to the execution unit program of the allocation request source that has stopped the operation. The operation of the execution unit program of the allocation request source that has been assigned is resumed, and at the same time, the operation unit program that has resumed the operation is caused to function as a management unit that assigns the storage resource requested to be assigned.

  The image processing program according to the invention described in claim 17 is a program for causing a computer having a storage resource and a program execution resource to function as the image processing unit and the management means. By executing the image processing program according to the present invention, the computer functions as the image processing apparatus according to claim 1, and, similarly to the invention according to claim 1, the pipeline form or the directed non-direction Among a plurality of image processing modules connected in the form of a cyclic graph, in a configuration for executing image processing by executing a plurality of execution unit programs corresponding to different image processing modules in parallel, storage resources are temporarily insufficient. In addition, the entire image processing can be continued.

  As described above, the present invention determines whether or not a storage resource requested for allocation can be allocated every time storage resource allocation is requested from an arbitrary execution unit program. The operation of the execution unit program of the allocation request source is stopped and the operation of the execution unit program of the allocation request source that stopped the operation is restarted when it is determined that the storage resource requested to be allocated can be allocated. Since the storage resource is allocated to the execution unit program of the allocation request source whose operation has been resumed, among the plurality of image processing modules connected in a pipeline form or a directed acyclic graph form, different image processing modules In a configuration that performs image processing by executing multiple execution unit programs corresponding to the It is possible also to continue the whole image processing and, has an excellent effect that.

  Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a computer 10 capable of functioning as an image processing apparatus according to the present invention. The computer 10 is incorporated in any image handling device that needs to perform image processing internally, such as a copying machine, a printer, a facsimile machine, a multifunction machine having these functions, a scanner, a photographic printer, or the like. Alternatively, it may be an independent computer such as a personal computer (PC), or may be a computer incorporated in a portable device such as a PDA (Personal Digital Assistant) or a cellular phone.

  The computer 10 includes a CPU 12, a memory 14, a display unit 16, an operation unit 18, a storage unit 20, an image data supply unit 22 and an image output unit 24, which are connected to each other via a bus 26. When the computer 10 is incorporated in an image handling device as described above, a display panel, a numeric keypad, or the like composed of an LCD or the like provided in the image handling device can be applied as the display unit 16 and the operation unit 18. When the computer 10 is an independent computer, a display, a keyboard, a mouse, or the like connected to the computer can be applied as the display unit 16 or the operation unit 18. The storage unit 20 is preferably an HDD (Hard Disk Drive), but other nonvolatile storage means such as a flash memory can be used instead.

  The image data supply unit 22 only needs to be able to supply image data to be processed. For example, an image reading unit that reads an image recorded on a recording material such as paper or photographic film, and outputs the image data. A receiving unit that receives image data from the outside via a line, an image storage unit (memory 14 or storage unit 20) that stores image data, and the like can be applied. The image output unit 24 may be any unit that outputs image data that has undergone image processing or an image represented by the image data. For example, the image recording unit records an image represented by the image data on a recording material such as paper or a photosensitive material. A display unit that displays an image represented by image data on a display, a writing device that writes image data to a recording medium, and a transmission unit that transmits image data via a communication line can be applied. The image output unit 24 may be an image storage unit (memory 14 or storage unit 20) that simply stores image data that has undergone image processing.

  As shown in FIG. 1, the storage unit 20 includes various programs executed by the CPU 12, such as management of resources such as the memory 14, management of execution of programs by the CPU 12, and communication between the computer 10 and the outside. A program of the system 30, an image processing program group 34 for causing the computer 10 to function as an image processing apparatus according to the present invention, and a desired image for the image processing apparatus realized by the CPU 12 executing the image processing program group. Various programs of applications 32 (indicated as application program group 32 in FIG. 1) to be processed are stored.

  The image processing program group 34 reduces the development load when developing the above-described various image handling devices and portable devices, and reduces the development load when developing an image processing program usable on a PC or the like. The purpose is a program developed so that it can be commonly used in various devices (platforms) such as various image handling devices, portable devices, and PCs, and corresponds to the image processing program according to the present invention. The image processing apparatus realized by the image processing program group 34 constructs an image processing unit that performs image processing instructed by the application 32 in accordance with a construction instruction from the application 32, and executes the image processing unit in accordance with an execution instruction from the application 32. The image processing program group 34 instructs the construction of an image processing unit (an image processing unit having a desired configuration) for performing desired image processing, or the constructed image processing unit. The application 32 is provided with an interface for instructing the execution of the image processing by. For this reason, even when newly developing an arbitrary device that needs to perform image processing internally, regarding the development of a program for performing the image processing, the above-described interface is used to perform the image processing required for the device. It is only necessary to develop the application 32 to be used and executed by the image processing program group 34, and it is not necessary to newly develop a program for actually performing image processing, so that the development load can be reduced.

  Further, as described above, the image processing apparatus realized by the image processing program group 34 constructs an image processing unit that performs image processing instructed by the application 32 in accordance with the construction instruction from the application 32, and constructs the constructed image processing unit. Therefore, for example, when the color space of the image data to be processed and the number of bits per pixel are indefinite, or the contents, procedures, parameters, etc. of the image processing to be executed are indefinite, When the application 32 instructs to reconstruct the image processing unit, the image processing executed by the image processing apparatus (image processing unit) can be flexibly changed according to the image data to be processed.

  Hereinafter, the image processing program group 34 will be described. As shown in FIG. 1, the image processing program group 34 is roughly divided into a module library 36, a process construction unit 42 program, and a process management unit 46 program. Although details will be described later, the processing construction unit 42 according to the present embodiment, according to an instruction from the application, as illustrated in FIG. 2 as an example, one or more image processing modules 38 that perform predetermined image processing, A buffer module 40 that is arranged in at least one of the preceding stage and the subsequent stage of each image processing module 38 and has a buffer for storing image data, and a pipeline form or a DAG (Directed Acyclic Graph) form An image processing unit 50 formed by linking is constructed. The entity of each image processing module constituting the image processing unit 50 is a first program executed by the CPU 12 and causing the CPU 12 to perform predetermined image processing, or an external unit that is executed by the CPU 12 and not shown in FIG. 2 is a second program for instructing the execution of processing on the image processing apparatus (for example, a dedicated image processing board), and the module library 36 described above stores different predetermined image processing (for example, input processing and filter processing). , Color conversion processing, enlargement / reduction processing, skew angle detection processing, image rotation processing, image composition processing, output processing, and the like) are registered. Hereinafter, in order to simplify the description, it is assumed that the individual image processing modules constituting the image processing unit 50 are the first program.

  As shown in FIG. 3A as an example, each image processing module 38 includes an image processing engine 38A that performs image processing on image data by a predetermined unit processing data amount, and a front stage and a rear stage of the image processing module 38. The control unit 38B is configured to input / output image data to / from the module and to control the image processing engine 38A. The unit processing data amount in each image processing module 38 is the image processing engine 38A from an arbitrary number of bytes including one line of the image, a plurality of lines of the image, one pixel of the image, one surface of the image, and the like. Is selected and set in advance according to the type of image processing to be performed. For example, in the image processing module 38 that performs color conversion processing and filter processing, the unit processing data amount is one pixel, and the image to be enlarged / reduced is processed. In the processing module 38, the unit processing data amount is equivalent to one line of the image or a plurality of lines of the image, and in the image processing module 38 that performs image rotation processing, the unit processing data amount is equivalent to one image, and the image compression / decompression processing is performed. In the image processing module 38 to be performed, the unit processing data amount is set to N bytes depending on the execution environment.

  The module library 36 also registers image processing modules 38 having the same type of image processing executed by the image processing engine 38A and different contents of the image processing executed (in FIG. 1, this type of image processing is performed). Modules are indicated as “module 1” and “module 2”). For example, with respect to the image processing module 38 that performs enlargement / reduction processing, the image processing module 38 that performs reduction processing to reduce the input image data to 50% by thinning out every other pixel is designated for the input image data. A plurality of image processing modules 38 such as an image processing module 38 that performs enlargement / reduction processing at the enlarged / reduced rate are prepared. For example, the image processing module 38 that performs color conversion processing includes an image processing module 38 that converts the RGB color space to the CMY color space, an image processing module 38 that converts the color space to the opposite, and an L * a * b * color space. Image processing modules 38 for performing other color space conversions are prepared.

  In addition, the control unit 38B of the image processing module 38 receives an image from a module (for example, the buffer module 40) in the previous stage of its own module in order to input image data necessary for the image processing engine 38A to process each unit processing data amount. The data is acquired for each unit read data amount, and the image data output from the image processing engine 38A is output to the subsequent module (for example, the buffer module 40) for each unit write data (the amount of data such as compression by the image processing engine 38A) If the image processing with increase / decrease is not performed, the unit writing data amount = the unit processing data amount) or the result of the image processing by the image processing engine 38A is output to the outside of the module (for example, the image processing engine 38A is skewed). When image analysis processing such as corner detection processing is performed, a skew is used instead of image data. Image analysis processing results such as corner detection results may be output), but the module library 36 has the same type and content of image processing executed by the image processing engine 38A, and the above unit processing data Image processing modules 38 having different amounts, unit read data amounts, and unit write data amounts are also registered. For example, the unit processing data amount in the image processing module 38 that performs the image rotation processing has been described as one image. An image corresponding to a plurality of lines may be included in the module library 36.

  The program of each image processing module 38 registered in the module library 36 is composed of a program corresponding to the image processing engine 38A and a program corresponding to the control unit 38B, but a program corresponding to the control unit 38B. The image processing module 38 having the same unit read data amount and unit write data amount among the individual image processing modules 38 is concerned with the type and content of image processing executed by the image processing engine 38A. However, the program corresponding to the control unit 38B is shared (the same program is used as the program corresponding to the control unit 38B). Thereby, the development load in developing the program of the image processing module 38 is reduced.

  In the image processing module 38, the unit read data amount and the unit write data amount are not fixed when the input image attribute is unknown, and the input image data attribute is acquired and acquired. There is a module in which the unit read data amount and the unit write data amount are determined by substituting the attribute into a predetermined arithmetic expression, and this type of image processing module 38 has a unit read data amount and For the image processing module 38 in which the unit write data amount is derived using the same arithmetic expression, a program corresponding to the control unit 38B may be shared. The image processing program group 34 according to the present embodiment can be mounted on various devices as described above, but the number, type, and the like of the image processing modules 38 registered in the module library 36 in the image processing program group 34. Needless to say, addition, deletion, replacement, and the like can be appropriately performed in accordance with image processing required by various devices that implement the image processing program group 34.

  Each buffer module 40 constituting the image processing unit 50 is configured by a memory area secured through the operating system 30 from the memory 14 provided in the computer 10 as shown in FIG. 3B as an example. And a buffer control unit 40B that performs input / output of image data between the front and rear modules of the buffer module 40 and management of the buffer 40A. The buffer control unit 40B of each buffer module 40 is also a program executed by the CPU 12, and the program of the buffer control unit 40B is also registered in the module library 36 (in FIG. 1, the program of the buffer control unit 40B is registered). "Buffer module")

  Further, the processing construction unit 42 that constructs the image processing unit 50 in accordance with an instruction from the application 32 includes a plurality of types of module generation units 44 as shown in FIG. The plural types of module generation units 44 support different image processing, and are activated by the application 32 to generate a module group including an image processing module 38 and a buffer module 40 for realizing the corresponding image processing. Perform the process. In FIG. 1, the module generation unit 44 corresponding to the type of image processing executed by each image processing module 38 registered in the module library 36 is shown as an example of the module generation unit 44. The image processing corresponding to the generation unit 44 may be image processing realized by a plurality of types of image processing modules 38 (for example, skew correction processing including skew angle detection processing and image rotation processing). When the required image processing is processing combining a plurality of types of image processing, the application 32 sequentially activates the module generation unit 44 corresponding to any of the plurality of types of image processing. As a result, an image processing unit 50 that performs necessary image processing is constructed by the module generation unit 44 that is sequentially activated by the application 32.

  As shown in FIG. 1, the processing management unit 46 includes a workflow management unit 46 </ b> A that controls execution of image processing in the image processing unit 50, the memory 14 by each module of the image processing unit 50, and various files of the computer 10. It includes a resource management unit 46B that manages the use of resources, and an error management unit 46C that manages errors that have occurred in the image processing unit 50. In the present embodiment, the image processing unit 50 constructed by the processing construction unit 42 has the individual image processing modules 38 constituting the image processing unit 50 moved to the subsequent stage with a data amount smaller than one image as a unit. It operates to perform image processing in parallel while delivering image data (referred to as block unit processing).

  The error management unit 46C acquires error information such as the type and location of the error that has occurred when an error occurs while the image processing unit 50 is executing image processing, and the image processing program group 34 Is obtained from the storage unit 20 or the like, and an error notification method according to the device environment represented by the obtained device environment information is determined. A process for notifying the occurrence of an error by the determined error notification method is performed.

  Next, the operation of this embodiment will be described. In a device in which the image processing program group 34 is installed, when a situation where it is necessary to perform some kind of image processing, this situation is detected by a specific application 32. In addition, as a situation where it is necessary to perform image processing, for example, an image is read by an image reading unit as the image data supply unit 22, and is recorded as an image on a recording material by an image recording unit as the image output unit 24, or image output Whether the image is displayed on the display unit as the unit 24, the image data is written to the recording medium by the writing device as the image output unit 24, the image data is transmitted by the transmission unit as the image output unit 24, or the image output When the execution of a job to be stored in the image storage unit as the unit 24 is instructed by the user, or received by the reception unit as the image data supply unit 22 or stored in the image storage unit as the image data supply unit 22 For the image data being recorded, recording on the recording material, display on the display unit, writing to the recording medium, transmission, Any execution of a job for the storage of the image storage unit include when instructed by the user. In addition, the situation where image processing needs to be performed is not limited to the above. For example, in a state where the names of processes that can be executed by the application 32 in accordance with an instruction from the user are displayed on the display unit 16 as a list, For example, the execution target process may be selected by the user.

  As described above, when detecting that it is necessary to perform some kind of image processing, the application 32 first recognizes the type of the image data supply unit 22 that supplies the image data to be processed, and recognizes the recognized type. Is a buffer area (partial area of the memory 14), a parameter for causing the buffer control unit 40B to recognize the buffer area designated as the image data supply unit 22 as an already secured buffer 40A is set, and the buffer By generating a thread for executing the program of the control unit 40B (generating the buffer control unit 40B), the buffer module 40 including the designated buffer area (the buffer module 40 functioning as the image data supply unit 22) is generated. . The above thread corresponds to the execution unit program according to the present invention. Further, the buffer module 40 may be generated as a process or an object instead of a thread.

  Subsequently, in the same manner as described above, the application 32 recognizes the type of the image output unit 24 as the output destination of the image data subjected to image processing, and the recognized type is the buffer area (partial area of the memory 14). In this case, the buffer module 40 including the buffer area designated as the image output unit 24 is generated in the same manner as described above. The buffer module 40 generated here functions as the image output unit 24. Further, the application 32 recognizes the contents of the image processing to be executed, decomposes the image processing to be executed into a combination of image processing at a level corresponding to each module generation unit 44, and realizes the image processing to be executed. Therefore, the type of image processing necessary for the purpose and the execution order of the individual image processing are determined. In this determination, for example, the type of image processing and the execution order of the individual image processing are registered in advance as information in association with the type of job that can be instructed by the user. Can be realized by reading out information corresponding to the type of job instructed.

  Then, the application 32 activates the module generation unit 44 corresponding to the specific image processing based on the image processing type and execution order determined above (generates a process, thread, or object for executing the program of the module generation unit 44). ), The input module identification information for identifying the input module for inputting image data to the module group, as information necessary for generating the module group by the module generation unit 44, Output module identification information for identifying an output module from which the module group outputs image data, input image attribute information indicating an attribute of input image data input to the module group, and parameters of image processing to be executed To generate a corresponding module group. Further, when the required image processing is a combination of a plurality of types of image processing, the application 32 responds to individual image processing when notified of completion of generation of the module group from the instructed module generation unit 44. The process of starting the other module generation unit 44 and notifying the information necessary for generating the module group is repeated in ascending order of the execution order of the individual image processes.

  In the above input module, the image data supply unit 22 is the input module for the module group with the first execution order, and the last module (usually the buffer) for the module group with the second execution order or later. Module 40) is the input module. As for the above output modules, the image output unit 24 is designated as the output module in the module group whose execution order is the last, so that the image output unit 24 is designated as the output module. Specification by the application 32 is not performed for confirmation, and the module generation unit 44 generates and sets it when necessary. The input image attributes and image processing parameters are registered as information in advance in association with, for example, the type of job that can be instructed by the user, and information corresponding to the type of job instructed to be executed is read out. As a result, the application 32 may recognize it, or the user may specify it.

  On the other hand, the module generation unit 44 performs module generation processing when activated by the application 32. In the module generation process, first, input image attribute information representing an attribute of input image data input to the generation target image processing module 38 is acquired. Note that the processing for acquiring the attributes of the input image data is performed in the case where the buffer module 40 exists in the preceding stage of the image processing module 38 to be generated, and the image processing module in the preceding stage that writes image data in the buffer module 40. This can be realized by acquiring the attributes of the output image data from the image data 38.

  Then, based on the attribute of the input image data represented by the acquired information, it is determined whether the generation of the image processing module 38 to be generated is necessary. For example, the module generation unit 44 is a module generation unit that generates a group of modules for performing color conversion processing. When the CMY color space is specified by the application 32 as the color space of the output image data by the image processing parameter, the acquired input image When the input image data is found to be RGB color space data based on the attribute information, it is necessary to generate an image processing module 38 that performs RGB → CMY color space conversion as an image processing module 38 that performs color space processing. However, when the input image data is data in the CMY color space, the attribute of the input image data and the attribute of the output image data match with respect to the color space, so the image processing module 38 that performs the color space conversion process generates Judge as unnecessary.

  If it is determined that the generation target image processing module 38 needs to be generated, it is determined whether the buffer module 40 is required after the generation target image processing module 38. This determination is made when the subsequent stage of the image processing module is the output module (image output unit 24) (see, for example, the last stage image processing module 38 in the image processing unit 50 shown in FIGS. 2A to 2C). As an example, like the image processing module 38 that performs skew angle detection processing in the image processing unit 50 shown in FIG. In the case other than the above, the determination is affirmed and the buffer control unit 40B is activated (a thread for executing the program of the buffer control unit 40B is generated). Thus, the buffer module 40 connected to the subsequent stage of the image processing module 38 is generated. The above thread also corresponds to the execution unit program according to the present invention. Further, the buffer module 40 may be generated as a process or an object instead of a thread.

  Subsequently, information on the preceding module (for example, the buffer module 40), information on the succeeding buffer module 40 (only the image processing module 38 that generated the buffer module 40 in the succeeding stage), and input image data input to the image processing module 38 Attributes and processing parameters are given and registered in the module library 36, and from among a plurality of candidate modules that can be used as the image processing module 38, the attributes of the input image data acquired earlier and the image processing module 38 The image processing module 38 that matches the processing parameter to be executed is selected and generated (a thread for executing the program of the image processing engine 38A and the control unit 38B is generated). The above thread also corresponds to the execution unit program according to the present invention. Further, the image processing module 38 may be generated as a process or an object instead of a thread.

  For example, the module generation unit 44 is a module generation unit that generates a group of modules that perform color conversion processing. The CMY color space is designated as the color space of the output image data by the processing parameters, and the input image data is data in the RGB color space. If there is, an image processing module 38 that performs RGB → CMY color space conversion is selected and generated from a plurality of types of image processing modules 38 that perform various color space processing registered in the module library 36. The The image processing module 38 is an image processing module 38 that performs enlargement / reduction processing. If the designated enlargement / reduction ratio is other than 50%, the input image data is enlarged at the designated enlargement / reduction ratio. If the image processing module 38 that performs the reduction process is selected and generated and the designated enlargement / reduction ratio is 50%, the enlargement / reduction process specialized for the enlargement / reduction ratio of 50%, that is, the input image data is one pixel. An image processing module 38 that performs a reduction process to reduce to 50% by thinning out at intervals is selected and generated.

  The selection of the image processing module 38 is not limited to the above. For example, a plurality of image processing modules 38 having different unit processing data amounts in image processing by the image processing engine 38A are registered in the module library 36, and the image processing unit The image processing module 38 having an appropriate unit processing data amount is selected according to the operating environment such as the size of the memory area that can be allocated to 50 (for example, the image processing module 38 having a smaller unit processing data amount as the size decreases). Or the application 32 or the user may select it.

  When the generation of the image processing module 38 is completed, the workflow management unit 46A is notified of the set of the ID of the subsequent buffer module 40 and the ID of the generated image processing module 38. This ID may be information that can uniquely identify each module, and may be, for example, a number given in the order of generation of each module, an address of an object of the buffer module 40 or the image processing module 38 on a memory, or the like. Further, the module generation unit 44 performs image processing realized by a plurality of types of image processing modules 38 (for example, skew correction processing realized by the image processing module 38 that performs skew angle detection processing and the image processing module 38 that performs image rotation processing). When generating a module group for performing the above, the above process is repeated to generate a module group including two or more image processing modules 38. The module generation processing described above is sequentially performed by the individual module generation units 44 that are sequentially activated by the application 32, so that necessary image processing is performed as shown in FIGS. 2A to 2C, for example. An image processing unit 50 to perform is constructed.

  In the present embodiment, when the execution frequency of specific image processing is high, for example, the application 32 generates a plurality of types of module generation units 44 for generating the image processing unit 50 that performs specific image processing. Even after the image processing unit 50 that performs specific image processing is generated, it is left as a thread (may be a process or an object) by not instructing the end of processing, and each time it is necessary to perform specific image processing, It is also possible to regenerate the image processing unit 50 that performs specific image processing by sequentially instructing the remaining module generation units 44 to generate module groups. This eliminates the need to start each of the corresponding module generation units 44 every time specific image processing needs to be performed, and reduces the time required to regenerate the image processing unit 50 that performs specific image processing. can do.

  Incidentally, the control unit 38B of the image processing module 38 initializes the image processing module 38 when activated by the module generation unit 44. In this initialization, first, the information of the modules before and after the own module given from the module generation unit 44 is stored. Subsequently, the module preceding the own module is determined. If no module exists in the previous stage of the own module, no processing is performed. However, if the previous module is other than the buffer module 40, for example, the image data supply unit 22 or a specific file, as necessary. The initialization process is performed. When the buffer module 40 exists in the previous stage of the own module, the amount of image data (unit read data quantity) acquired by reading the image data from the previous stage buffer module 40 once is recognized. . This unit read data amount is only one if the number of buffer modules 40 in the previous stage of the own module is one. For example, the image processing unit 50 performs image composition processing in the image processing unit 50 shown in FIG. When there are a plurality of buffer modules 40 in the previous stage and the image processing engine 38A performs image processing using the image data respectively acquired from the plurality of buffer modules 40 as in the module 38, the individual buffer modules in the previous stage The unit read data amount corresponding to 40 is determined according to the type and content of image processing performed by the image processing engine 38A of the own module, the number of buffer modules 40 in the previous stage, and the like. Then, the unit read data amount is set in all the buffer modules 40 existing in the preceding stage by notifying the recognized unit read data amount to all the buffer modules 40 existing in the preceding stage (FIG. 3 ( (See also A) (1)).

  Next, the module following the own module is determined. If the module subsequent to the own module is other than the buffer module 40, such as the image output unit 24 or a specific file, the initialization process is performed as necessary (for example, if the subsequent module is the image output unit 24, the unit For example, processing for notifying output of image data by the amount of data corresponding to the amount of write data). If the subsequent module is the buffer module 40, the amount of image data (unit write data amount) in one image data write is recognized, and the unit write data amount is set in the subsequent buffer module. (See also (2) in FIG. 3A). Then, the completion of initialization of the image processing module 38 is notified to the module generation unit 44, and the process is terminated.

  On the other hand, the buffer control unit 40B of each buffer module 40 constituting the image processing unit 50 initializes the buffer module 40 when activated by the module generation unit 44 or the application 32. When the buffer module 40 is initialized, first, the unit write data amount is notified from the image processing module 38 in the previous stage of the own module or the unit read data amount is notified from the image processing module 38 in the subsequent stage of the own module. The notified unit write data amount or unit read data amount is stored (see also (1) and (2) in FIG. 3B).

  When the unit write data amount or the unit read data amount is notified from all the image processing modules 38 connected to the own module, the unit documents set by the individual image processing modules 38 connected to the own module respectively. The size of the unit buffer area, which is the management unit of the buffer 40A of the own module, is determined based on the embedded data amount and the unit read data amount, and the determined size of the unit buffer area is stored. As the size of the unit buffer area, the maximum value of the unit write data amount and the unit read data amount set in the own module is preferable, but the unit write data amount may be set or the unit read data amount may be set. The amount of data (the maximum value of the unit read data amount set by each image processing module 38 when a plurality of image processing modules 38 are connected to the subsequent stage of the own module) may be set. The least common multiple of the write data amount and the unit read data amount (the maximum value thereof) may be set. If the least common multiple is less than the predetermined value, the least common multiple is set. A value (for example, any one of the above-mentioned unit write data amount and unit read data amount, the unit write data amount, and the unit read data amount (the maximum value)) may be set.

  Further, when the own module is generated by the application 32 and is the buffer module 40 functioning as the image data supply unit 22 or the image output unit 24, the memory area used as the buffer 40A of the own module already exists. The size of the unit buffer area determined in advance is changed to the size of the existing memory area used as the buffer 40A of the own module. Further, valid data pointers corresponding to the individual image processing modules 38 subsequent to the module are generated, and the generated valid data pointers are initialized. This valid data pointer is the image data (valid data) that has not been read by the corresponding subsequent image processing module 38 among the image data written in the buffer 40A of the own module by the preceding image processing module of the own module. Pointers that indicate the start position (next read start position) and the end position, respectively. At initialization, specific information that usually means that no valid data exists is set. In the case of the buffer module 40 that is generated by the application 32 and functions as the image data supply unit 22, image data to be image-processed may already be written in the memory area used as the buffer 40A of the own module. Indicates that the start and end positions of the image data are Each set of the valid data pointers corresponding to the individual image processing module 38. With the above processing, the initialization of the buffer module 40 is completed, and the buffer control unit 40B notifies the completion of initialization to the workflow management unit 46A.

  On the other hand, when the construction of the image processing unit 50 that performs the required image processing is completed by the above-described module generation processing being sequentially performed by the module generation unit 44 that is sequentially activated, the application 32 starts the program of the workflow management unit 46A. By starting a thread (or process or object) that executes the process, the image processing unit 50A is instructed to execute image processing by the workflow management unit 46A.

  The workflow management unit 46A of the process management unit 46 performs the block unit control process shown in FIG. 7 when the program is started. In the block unit processing, the workflow management unit 46A inputs a processing request to a predetermined image processing module 38 among the image processing modules 38 constituting the image processing unit 50, thereby performing image processing by the image processing unit 50 in block units. In the following description, the processing after the completion of the initialization process performed by the buffer control unit 40B of each buffer module 40, the control unit of each image processing module 38, before the description of the operation of the entire image processing unit 50 is performed. The image processing module control process performed by 38B will be described in order.

  In this embodiment, when the image processing module 38 writes image data to the subsequent buffer module 40, a write request is input from the image processing module 38 to the buffer module 40, and the image processing module 38 receives the previous buffer module 40. When reading image data from the image processing module 38, a read request is input from the image processing module 38 to the buffer module 40. When a write request from the preceding image processing module 38 is input to the buffer module 40 (and when a timer described later times out), the data writing process described below is executed by the buffer control unit 40B. .

  In the data writing process, first, it is determined whether or not the buffer 40A of the own module is being used by another thread. Since data is also read from the buffer 40A asynchronously with the data write, when the buffer 40A is in use, the input write request information is stored in a work memory or the like, and a timer is started to write the data. The process is temporarily terminated. In the subsequent processing, the input write request information is used as information to be processed. However, if the timer has timed out and the data write processing is started, it is input in the past and stored in the work memory or the like. The write request information being taken out is extracted from the work memory or the like, and the subsequent processing is performed using the taken write request information as information to be processed.

  Subsequently, in the data writing process, the unit management data amount is notified to the resource management unit 46B as the size of the memory area to be secured, and the memory area used for writing (the writing buffer area: FIG. 4B is also shown). Reference) is acquired via the resource management unit 46B (the processing in the resource management unit will be described later). Next, a unit buffer area (a unit buffer in which image data of the unit write data amount can be written) has an empty area larger than the unit write data amount in the storage unit buffer area constituting the buffer 40A of the own module. It is determined whether or not (region) exists. In the buffer module 40 generated by the module generation unit 44, a memory area (unit buffer area) used as the buffer 40A is not initially secured, and a unit buffer area is secured as a unit whenever a memory area shortage occurs. Therefore, when a write request is first input to the buffer module 40, there is no memory area (unit buffer area) used as the buffer 40A, and this determination is denied. Further, even after the unit buffer area used as the buffer 40A is secured through the processing described later, the free area in the unit buffer area is less than the unit write data amount as image data is written to the unit buffer area. The above determination is also denied in the case of

  If it is determined that there is no unit buffer area (unit buffer area into which image data of the unit write data amount can be written) that has an empty area greater than the unit write data amount, the size of the memory area to be secured (A size of the unit buffer area) is notified to the resource management unit 46B, and a memory area (a unit buffer area used for storing image data) used as the buffer 40A of the own module is acquired via the resource management unit 46B. Then, using the previously acquired write buffer area as the write area, the start address of the write area is notified to the image processing module 38 of the write request source, and the start address that has notified the image data to be written Request to write in order. As a result, the image processing module 38 of the writing request source writes the image data in the writing buffer area notified of the head address (see also FIG. 4B).

  For example, if the size of the unit buffer area is not an integral multiple of the unit write data amount, the writing of the unit write data amount of image data to the buffer 40A (unit buffer area) is repeated, for example, as shown in FIG. ), The size of the empty area in the unit buffer area with the empty area is smaller than the unit write data amount. In this case, the area in which the image data of the unit write data amount is written extends over a plurality of unit buffer areas, but in this embodiment, the memory area used as the buffer 40A is secured in units of the unit buffer area. It is not guaranteed that the unit buffer areas secured at different timings are continuous areas on the real memory (memory 14). On the other hand, in the present embodiment, the image processing module 38 writes the image data to the writing buffer area secured separately from the unit buffer area for storage, as shown in FIG. Since the image data once written in the write buffer area is copied to a single unit buffer area or a plurality of unit buffer areas for storage, regardless of whether the area in which the image data is written extends over a plurality of unit buffer areas. As described above, the notification of the write area to the image processing module 38 that is the write request source only needs to be notified of the start address, and the interface with the image processing module 38 is simplified.

  If the own module is the buffer module 40 generated by the application 32, that is, if the memory area used as the buffer 40A is already secured, the address of the already secured memory area is written in the image processing module 38. This is notified as the address of the embedded area, and the image data is written to the memory area. When the writing of the image data to the writing area by the image processing module 38 in the previous stage is completed, the attribute information is added to the image data written in the writing buffer area, and then written in the storage buffer area as it is. If the size of the empty area in the unit buffer area with the empty area is smaller than the unit write data amount, the image data written in the write buffer area is stored as shown in FIG. Are written separately in a plurality of unit buffer areas.

  Then, the pointer indicating the end position of the valid data among the valid data pointers corresponding to the individual image processing modules 38 subsequent to the own module is moved backward by the unit write data amount. At the same time as updating (see also FIG. 4C), the memory area previously secured as the write buffer area is released by the resource management unit 46B, and the data writing process is temporarily terminated. The write buffer area may be secured when the buffer module 40 is initialized and may be released when the buffer module 40 is erased.

  Subsequently, a data read process executed by the buffer control unit 40B of the buffer module 40 when a read request is input from the subsequent image processing module 38 to the buffer module 40 (and a timer described later times out). explain.

  In the data reading process, first, in step 170, it is determined whether or not the activation factor of the current data reading process is the activation by the input of a reading request from the subsequent image processing module. Read request information input from the processing module is registered at the end of the read queue. Next, it is determined whether or not the buffer 40A of the own module is being used by another thread. If the buffer 40A is in use, it is determined whether or not the read request information is registered in the read queue. If the read request information is not registered, the data read process is terminated and the read request information is registered. If so, the data read processing is terminated after starting the timer. When this timer times out, the data reading process is started again, an unprocessed read request (information) registered in the read queue is taken out again, and a process corresponding to the read request is performed.

  On the other hand, if the buffer 40A of the own module is not in use, the read request information registered at the head is extracted from the read queue, and the read request source is based on the request source identification information included in the read request information. The image processing module 38 is recognized, the unit read data amount set by the image processing module 38 of the read request source is recognized, and the read request is based on the valid data pointer corresponding to the image processing module 38 of the read request source. The beginning position and the end position of the valid data on the buffer 40A corresponding to the original image processing module 38 are recognized. Next, based on the recognized start position and end position of valid data, valid data corresponding to the read request source image processing module 38 (image data that can be read by the read request source image processing module 38) is unit read data. Judge whether there is more than the amount.

  If the valid data corresponding to the image processing module 38 of the read request source is less than the unit read data amount, whether or not the end of the valid data that can be read by the image processing module 38 of the read request source is the end of the image data to be processed judge. Valid data corresponding to the image processing module 38 of the read request source is stored in the buffer 40A at a unit read data amount or more, or valid data corresponding to the image processing module 38 of the read request source stored in the buffer 40A is stored. If the end of the valid data is the end of the image data to be processed, although it is less than the unit read data amount, the unit read corresponding to the image processing module 38 that is the read request source is the size of the memory area to be secured. In addition to notifying the resource management unit 46B of the amount of data, the resource management unit 46B is requested to secure a memory area used for reading (read buffer area: see also FIG. 5B), and read via the resource management unit 46B. Get the buffer area.

  Next, the effective data to be read is read from the buffer 40A by the amount of the unit read data and written to the read buffer area, and the head address of the read buffer area is used as the read area start address to the read request source image processing module 38. Along with the notification, the image data is requested to be read sequentially from the notified head address. As a result, the image processing module 38 as the read request source reads the image data from the read area (read buffer area) to which the head address is notified. When the effective data to be read is data corresponding to the end of the image data to be processed, when the image data is requested to be read, the size of the image data to be read and the size of the image data to be read are included at the end of the image data to be processed. This is also notified to the image processing module 38 of the reading request source. When the own module is the buffer module 40 generated by the application 32, the memory area (collection of unit buffer areas) used as the buffer 40A is a continuous area. The writing of the target image data to the reading buffer area may be omitted, and the subsequent image processing module 38 may read the image data directly from the unit buffer area.

  As an example, as shown in FIG. 5A, the amount of valid data stored in the unit buffer area in which the image data of the head portion of the valid data is stored is less than the unit read data amount. When the target valid data extends over a plurality of unit buffer areas, the valid data to be read this time is not necessarily stored in a continuous area on the real memory (memory 14). In the data reading process, as shown in FIGS. 5B and 5C, even in such a case, the image data to be read is once written in the reading buffer area and then read from the reading buffer area. Therefore, regardless of whether or not the image data to be read is stored across a plurality of unit buffer areas, the notification of the read area to the image processing module 38 of the read request is made as described above. It is only necessary to notify the start address, and the interface with the image processing module 38 is simplified.

  When the completion of reading of the image data from the reading area by the image processing module 38 of the reading request is notified, the start address and size of the memory area secured as the reading buffer area are notified to the resource management unit 46B, and the memory The area is released by the resource management unit 46B. This buffer area for reading may also be secured when the buffer module 40 is initialized and released when the buffer module 40 is erased. Further, by moving the pointer representing the start position of the valid data among the valid data pointers corresponding to the image processing module 38 that is the read request source, the start position of the valid data indicated by the pointer is moved backward by the unit read data amount. Update (see also FIG. 5C).

  Next, the effective data pointers corresponding to the individual image processing modules 38 in the subsequent stage are referred to, and the subsequent stages of the image data stored in the unit buffer area constituting the buffer 40A are updated by the previous pointer update. It is determined whether or not a unit buffer area that has been completely read by the image processing module 38, that is, a unit buffer area that does not store valid data, has appeared. If the determination is negative, the data reading process is completed through the above-described reading queue check process (determination of whether or not the read request information is registered in the reading queue), but the valid data When a unit buffer area that does not store is appeared, the unit buffer area is released by the resource management unit 46B, and then the data reading process is terminated through a check process for the reading queue.

  On the other hand, the amount of valid data stored in the buffer 40A and readable by the image processing module 38 that is the read request source is less than the unit read data amount, and the end of the readable valid data is the image data to be processed. If it is not the end (when no valid data that can be read is detected in (4) of FIG. 3B), a data request for requesting new image data is output to the workflow management unit 46A (FIG. 3 ( (See also (5) of B)) Read request information retrieved from the read queue is re-registered in the original queue (at the beginning or end), and then the data is read through the read queue check process. The process ends. In this case, the workflow management unit 46A inputs a processing request to the image processing module 38 in the previous stage of the own module. As a result, until the amount of valid data that can be read exceeds the unit read data amount or until the end of the valid data that can be read is detected to be the end of the image data to be processed, the corresponding readout is performed. The request information is saved in a read queue and is periodically retrieved to repeatedly perform the requested process.

  Although details will be described later, when a data request is input from the buffer module 40, the workflow management unit 46A inputs the processing request to the image processing module 38 in the preceding stage of the buffer module 40 that is the data request source (see FIG. 3B). (See also (6)). When the processing of the control unit 38B of the preceding image processing module 38 is triggered by the input of this processing request, when the preceding image processing module 38 can write image data to the buffer module 40, the preceding image processing is performed. When the writing request is input from the module 38, the above-described data writing process is performed, and the image data is written from the preceding image processing module 38 to the buffer 40A of the buffer module 40 ((7) in FIG. 3B). ), See also (8)). As a result, the image data is read from the buffer 40A by the subsequent image processing module 38 (see also (9) in FIG. 3B).

  In the above-described data writing process and data reading process, exclusive control is performed so that while one is accessing the buffer 40A of its own module, the other stops accessing the buffer 40A. Thereby, even if the CPU 12 of the computer 10 executes the threads corresponding to the individual modules constituting the image processing unit 50 in parallel, a plurality of requests are input to the single buffer module 40 simultaneously or substantially simultaneously. Since inconvenience can be avoided, the CPU 12 of the computer 10 can execute threads corresponding to individual modules in parallel.

  Subsequently, each time a processing request is input from the workflow management unit 46A to the individual image processing modules 38 constituting the image processing unit 50, the image processing modules respectively performed by the control unit 38B of the individual image processing module 38. The control process (FIG. 6) will be described.

  In the image processing module control processing, first, in step 219, the size of the memory used by the own module and the presence / absence of other resources used by the own module based on the type and contents of the image processing performed by the image processing engine 38A of the own module. Recognize Note that the memory used by the image processing module 38 is mainly a memory necessary for the image processing engine 38A to perform image processing. However, when the former module is the image data supply unit 22, or the latter module is an image. In the case of the output unit 24, there may be a need for a buffer memory for temporarily storing the image data when the image data is transmitted to or received from the preceding or succeeding module. Further, when information such as a table is included in the processing parameter, a memory area for holding it may be required. Then, the resource management unit 46B is requested to secure a memory area of the recognized size, and the memory area secured by the resource management unit 46B is acquired from the resource management unit 46B. When the own module (its image processing engine 38A) recognizes that it needs other resources than the memory, it requests the resource management unit 46B to secure the other resources, and allocates the other resources to the resource. Obtained from the management unit 46B.

  In the next step 220, when a module (buffer module 40, image data supply unit 22, image processing module 38, etc.) exists in the previous stage of its own module, data (image data or analysis) is sent to the preceding module. Image processing result). In the next step 222, it is determined whether data can be acquired from the preceding module. If the determination in step 222 is negative, it is determined in step 224 whether the end of the entire process has been notified. If the determination in step 224 is negative, the process returns to step 222, and steps 222 and 224 are repeated until data can be acquired from the preceding module. If the determination in step 222 is affirmative, data acquisition processing for acquiring data from the previous module in step 226 and writing the acquired data in the memory area for temporary storage of data in the memory area acquired in step 219 I do.

  Here, if the previous module of the own module is the buffer module 40, when data is requested in the previous step 220 (read request), the readable valid data is transferred to the buffer 40A of the buffer module 40 in the unit read data amount. As long as the end of the valid data that has been stored or readable matches the end of the image data to be processed, the image processing module 38 in the previous stage of the buffer module 40 immediately returns to the state of the image data to be processed. After the image data is written in the buffer 40A of the buffer module 40, the state is changed to the above state, and then the start address of the reading area is notified from the buffer module 40 and the reading of the image data is requested. As a result, the determination at step 222 is affirmed and the routine proceeds to step 226, where the image data of the unit read data amount (or data amount less than that) is read from the read area in which the head address is notified from the preceding buffer module 40, Data acquisition processing to be written in the temporary storage memory area is performed (see also (3) in FIG. 3A).

  If the previous module of the own module is the image data supply unit 22, the image data supply unit 22 immediately notifies that the image data can be acquired when the data request is output in the previous step 220. As a result, the determination in step 222 is affirmed, and the process proceeds to step 226. Image data acquisition processing for acquiring the unit read data amount of image data from the preceding image data supply unit 22 and writing it in the temporary storage memory area is performed. Do. If the previous module of the own module is the image processing module 38, a data request (processing request) is output in the previous step 220. If the previous image processing module 38 is ready to execute image processing, the data request is written. Since it is notified that the data (image processing result) can be acquired by inputting the load request, the determination in step 222 is affirmed, and the process proceeds to step 226. The data acquisition process for writing the data output from the image processing module 38 in the previous stage to the temporary storage memory area is performed by notifying the address of the temporary storage memory area in which data is written and requesting the writing.

  In the next step 228, it is determined whether or not a plurality of modules are connected to the previous stage of the own module. If the determination is negative, the process proceeds to step 232 without performing any processing. If the determination is affirmative, the process proceeds to step 230, and whether or not data has been acquired from all modules connected in the preceding stage. To determine. If the determination in step 230 is negative, the process returns to step 220, and steps 220 to 230 are repeated until the determination in step 230 is positive. When all the data to be acquired from the previous module is obtained, the determination in step 228 is denied or the determination in step 230 is affirmed, and the process proceeds to step 232.

  In the next step 232, an area for data output is requested to the module subsequent to the module, and the determination is repeated until the data output area can be acquired in step 232 (until the start address of the data output area is notified). . If the subsequent module is the buffer module 40, the request for the data output area is made by outputting a write request to the buffer module 40. When the data output area (the write area in which the start address is notified from the buffer module 40 if the subsequent module is the buffer module 40) can be obtained (see also (4) in FIG. 3A), the next step In 236, a memory area for image processing by the image processing engine among the data acquired in the previous data acquisition process, the data output area acquired from the subsequent module (first address), and the memory area acquired in the previous step 219 ( Is input to the image processing engine 38A, and the input data is subjected to predetermined image processing using a memory area for image processing (see also (5) in FIG. 3A). At the same time, the processed data is written in the data output area (see also (6) in FIG. 3A). When the input of the unit read data amount to the image processing engine 38A is completed and all the data output from the image processing engine 38A is written in the data output area, the subsequent step 238 indicates that the output has been completed. Notify the module.

  The processing (unit processing) for the data of the unit processing data amount in the image processing module 38 is completed by the above steps 220 to 238. However, in the processing request input from the workflow management unit 46A to the image processing module 38, the workflow management unit The number of executions of unit processing may be specified by 46A. Therefore, in step 240, it is determined whether the number of executions of the unit process has reached the number of executions instructed by the input processing request. If the number of executions of the instructed unit process is 1, this determination is unconditionally affirmed. However, if the number of executions of the instructed unit process is 2 or more, the process returns to Step 220, and the determination in Step 240 is performed. Steps 220 to 240 are repeated until affirmative. If the determination in step 240 is affirmed, the process proceeds to step 242 to output a processing completion notification to the workflow management unit 46A, thereby notifying the workflow management unit 46A that the processing corresponding to the input processing request has been completed. Then, the image processing module control process ends.

  Further, when the processing image data is processed to the end by repeating the above-described processing every time a processing request is input from the workflow management unit 46A, the end of the processing target image data is notified from the preceding module. Thus, the determination in step 224 is affirmed, and the process proceeds to step 244. The image data to be processed (note that the image data to be processed is often image data for one page, but is image data for a plurality of pages. An overall process end notification that means that the process for (may be present) has ended is output to the workflow management unit 46A and the subsequent module. In the next step 246, a process for requesting release of all acquired resources and deleting the own module is performed, and the image processing module control process is terminated.

  On the other hand, when activated by the application 32, the workflow management unit 46A performs a block unit control process 1 shown in FIG. As described above, when the processing request is input to each image processing module 38 of the image processing unit 50 by the workflow management unit 46A, the number of executions of the unit processing can be specified. In step 500 of 1, the number of executions of unit processing specified by one processing request is determined for each image processing module 38. The number of executions of unit processing per processing request can be determined so that, for example, the number of processing requests input to individual image processing modules 38 during the processing of the entire image data to be processed is averaged. However, it may be determined according to other criteria. The process of the next step 502 will be described later. In the next step 504, a processing request is input to the last image processing module 38 in the image processing unit 50 (see also (1) in FIG. 8), and the block unit control processing 1 is terminated.

Here, in the image processing unit 50 shown in FIG. 8, when the process from the workflow management unit 46A to the image processing module 38 ₄ of the last stage request is input, the control unit 38B of the image processing module 38 ₄ preceding buffer module 40 _A read request is input to ₃ (see (2) in FIG. 8). At this time, since the image processing module 38 ₄ readable valid data in the buffer 40A of the buffer module 40 ₃ (image data) is not stored, the buffer controller 40B of the buffer module 40 ₃ data in the workflow management unit 46A A request is input (see (3) in FIG. 8).

Each time the data request is input from the buffer module 40, the workflow management unit 46A performs the block unit control process 2 shown in FIG. In this block unit control processing 2, in step 510, the image processing module 38 (here, the image processing module 38 ₃ ) of the preceding stage of the buffer module 40 (here, the buffer module 40 ₃ ) of the data request input is recognized and recognized. A processing request is input to the preceding image processing module 38 (see (4) in FIG. 8), and the processing ends.

Image control unit 38B of the processing modules 38 _3, the processing request is input to the input read request preceding the buffer module 40 ₂ (see (5) in FIG. 8), read in the buffer 40A of the buffer module 40 ₂ since possible image data is not stored, the buffer controller 40B of the buffer module 40 ₂ inputs the data request to the workflow management unit 46A (see (6) in FIG. 8). Workflow management unit 46A, even if the data request is input from the buffer module 40 _2, by performing the block unit control processing 2 described above again, enter the processing request to the image processing module 38 ₂ of the previous stage (FIG. 8 (7)), the control unit 38B of the image processing module 38 ₃ inputs the read request to the buffer module 40 ₁ of the front reference ((8 in FIG. 8)). Further, since the readable image data to the buffer 40A of the buffer module 40 ₁ is not stored, the buffer controller 40B of the buffer module 40 ₁ also inputs the data request to the workflow management unit 46A (in FIG. 8 (9) reference). Even when a data request is input from the buffer module 40 ₁ , the workflow management unit 46A performs the block unit control process 2 again to input a processing request to the preceding image processing module 38 ₁ (FIG. 8). (See (10)).

Here, since the previous module of the image processing module 38 ₁ is the image data supply unit 22, the control unit 38 B of the image processing module 38 ₁ inputs a data request to the image data supply unit 22, and thereby the image data supply unit 22. The image data of the unit read data amount is acquired from the image data 22 (see (11) in FIG. 8), and the image data obtained by the image processing engine 38A performing image processing on the acquired image data is stored in the subsequent buffer. write module 40 _first buffer 40A (see (12) in FIG. 8).

Further, the buffer control unit 40B of the buffer module 40 ₁ requests the image processing module 38 ₂ to read when valid data exceeding the unit read data amount that can be read by the subsequent image processing module 38 ₂ is written. the image processing module 38 _{and second} control unit 38B with the reads image data of the unit read data amount from the buffer module 40 ₁ of buffer 40A (see (13) in FIG. 8), the image processing engine on the acquired image data 38A is an image data obtained by performing image processing, written in the following buffer module 40 _second buffer 40A (see (14) in FIG. 8). The buffer control unit 40B of the buffer module 40 ₂ requests the image processing module 38 ₃ to read when valid data exceeding the unit read data amount that can be read by the subsequent image processing module 38 ₃ is written, and the image processing module 38 _3. the control unit 38B of the (see (15) in FIG. 8) reads out the image data of the unit read data amount from the buffer module 40 ₂ of the buffer 40A, the image processing engine 38A on the acquired image data to perform image processing the image data obtained by, written to the subsequent buffer module 40 _third buffer 40A (see (16) in FIG. 8).

Furthermore, the buffer controller 40B of the buffer module 40 ₃ may request a read to the image processing module 38 ₄ When the subsequent image processing module 38 ₄ readable unit read data amount or more valid data is written, which control unit 38B of the image processing module 38 ₄ with the reads the image data of the unit read data amount from the buffer module 40 _third buffer 40A (see (17) in FIG. 8), the image processing engine on the acquired image data The image data obtained by the image processing performed by 38A is output to the image output unit 24, which is a subsequent module (see (18) in FIG. 8).

  In addition, when the control unit 38B of each image processing module 38 completes the writing of the image data to the buffer 40A of the subsequent buffer module 40, the control unit 38B inputs a processing completion notification to the workflow management unit 46A. The workflow management unit 46A performs the block unit control processing 3 shown in FIG. 7C every time a processing completion notification is input from the image processing module 38. In this block unit control process 3, after the process request is input again to the image processing module 38 that is the process completion notification source in step 520, the block unit control process 3 is terminated after performing the processes in steps 522 to 526. Note that the processing of step 524 to step 528 of the block unit control processing 3 will be described later.

  In this way, every time processing completion is notified from any image processing module 38, the processing request is input again to the image processing module 38 that is the processing completion notification source, so that the image processing for the image data to be processed is individually performed. In the image processing module 38, the execution is performed in parallel in a block unit execution form. When the image data supplied from the image data supply unit 22 reaches the end of the image data to be processed, the input of the overall processing end notification from the individual image processing module 38 to the workflow management unit 46A is sent to the preceding image. The processing modules 38 are sequentially performed.

  The workflow management unit 46A performs the block unit control process 4 shown in FIG. 7D every time an overall process end notification is input from the image processing module 38. In this block unit control process 4, in step 540, it is determined whether the image processing module 38 that is the input source of the overall process end notification is the last-stage image processing module 38. If the determination is negative, the process ends without performing any process, but all the image data that has undergone the necessary image processing on the image data to be processed is output to the image output unit 24. When an overall processing end notification is input from the last-stage image processing module 38, the determination in step 540 is affirmed, the process proceeds to step 542, the application 32 is notified of completion of image processing, and block unit control is performed. Processing 4 ends. The application 32 notified of the completion of the image processing notifies the user of the completion of the image processing.

  By the way, the image processing unit according to this embodiment is constructed by connecting the image processing module 38 and the buffer module 40 in a pipeline form or a directed acyclic graph form. If image data larger than the unit read data amount is not stored in the connected buffer module 40, image processing in the own module cannot be started (except for the first image processing module 38 connected to the image data supply unit 22). The progress of the image processing in each image processing module 38 depends on the progress of the image processing in the image processing module 38 located in the preceding stage, and particularly at the start of execution of a series of image processing in the image processing unit. In the near period, each image processing module, pipeline form or directed acyclic Better to execute the image processing preferentially in an image processing module located in the preceding stage is improved performance in rough form.

  In the configuration in which the image processing module 38 and the buffer module 40 are connected in a pipeline form or a directed acyclic graph form, the image processing module 38 on the rear stage side performs image processing more than the image processing module 38 on the front stage side. Since the progress always follows and the remaining amount of image data to be processed is always greater in the image processing module 38 on the subsequent stage, it is positioned on the subsequent stage as the series of image processing progresses in the image processing unit. The processing efficiency is improved by increasing the execution priority of the image processing in the image processing module, and the entire processing is completed especially at the end of the execution of the series of image processing in the image processing unit or in the vicinity thereof. As the number of image processing modules 38 that have been gradually increased from the front side, the image processing module in the image processing module that is located on the rear side is superior. It is desirable from the viewpoint of treatment efficiency for higher degrees.

  Based on the above, the workflow management unit 46A according to the present embodiment performs the process of each image processing module 38 in step 502 of the block unit control process 1 (see FIG. 7A) executed when activated by the application 32. As shown in FIG. 9A as an example, the execution priority of each thread that executes a program is the position of the image processing module 38 in the connected form of the pipeline form or the directed acyclic graph form. The execution priority of the individual threads is initially set so as to increase according to

  Note that the “position of the image processing module 38” is assigned in ascending order from the top (frontmost) image processing module 38 as shown in FIG. 10A, for example, if the image processing unit is in a pipeline form. It can be determined on the basis of the position value (or the position value given in descending order from the last (last stage) image processing module 38), and if the image processing unit is in a directed acyclic graph form, FIG. ), Position values are assigned in ascending order (or descending order from the last (last stage) image processing module 38) from the top (frontmost) image processing module 38, and a plurality of image processings are performed via the buffer module. For the image processing module 38 (image processing module E in the example of FIG. 10B) that acquires image data from the module, the maximum position value assigned to the plurality of image processing modules in the previous stage (Or minimum value) designated by the position value based on, can be determined on the basis of the position value.

Further, increasing the execution priority of the corresponding thread as the position of the image processing module 38 in the connected form of the pipeline form or the directed acyclic graph form becomes the preceding stage, for example, the thread corresponding to the image processing module The execution priorities that can be set to 9 are 9 levels of 1 to 9, and the position values are assigned to the individual image processing modules 38 in ascending order with the initial value = 1 from the previous stage side. For threads corresponding to
Execution priority = 10− (position value) However, if execution priority <1, execution priority = 1
Or a specific monotonically decreasing function (for example, position) where the execution priority is “9” when the position value is the minimum value and the execution priority is “1” when the position value is the maximum value. The execution priority may be set using a function that linearly decreases the execution priority with respect to an increase in value. Thereby, at the time when a series of image processing is started in the image processing unit, the higher priority is given to the thread in the preceding stage where the position of the corresponding image processing module 38 in the connected form of the pipeline form or the directed acyclic graph form The CPU 12 can execute the image processing with high processing efficiency by effectively using the CPU 12.

  As for the execution priority of the thread corresponding to the buffer module 40, a fixed priority may be fixedly set, or the image processing module 38 connected to the preceding stage and the subsequent stage of each buffer module 40. The average value, maximum value, and minimum value of the execution priority of the threads corresponding to the thread are set, and the thread execution priority corresponding to each image processing module 38 according to the degree of progress of the image processing, as described below, , The execution priority of the thread corresponding to each buffer module 40 may be changed.

  Further, the workflow management unit 46A according to the present embodiment performs block unit control processing 3 (see FIG. 7C) executed every time a processing completion notification is input from the image processing module 38, in step 522, the image processing unit. The degree of progress of image processing as a whole is determined. For this determination, for example, when a processing completion notification is transmitted from the individual image processing module 38 to the workflow management unit 46A, the degree of progress information that can determine the degree of progress of the image processing in the individual image processing module 38 is also transmitted. As described above, each of the image processing modules 38 is configured, and each time the workflow management unit 46A receives a processing completion notification from each of the image processing modules 38, the workflow management unit 46A holds the received progress degree information (the same image). If the degree of progress information previously received from the processing module 38 is already held, the degree of progress information already held is overwritten with the newly received degree of progress information), and then the individual image processing modules 38 This can be done by aggregating the degree of progress of image processing for the entire image processing unit from the corresponding degree of progress information. That.

  The above-described progress degree information is preferably information with a minimal load on the image processing module 38 (the CPU 12 that executes the corresponding thread) for derivation. For example, the individual image processing modules 38 for the entire image data to be processed. Information indicating the ratio of processed image data (specifically, the ratio of the amount of data, the ratio of the number of lines, etc.) can be used. Further, information indicating the amount of processed image data and the number of lines is transmitted from the individual image processing module 38 as the progress degree information, and the progress degree of the image processing in the individual image processing module 38 (the above ratio, etc.) The calculation may be performed by the workflow management unit 46A.

  In the next step 524, it is determined whether or not the degree of progress of the image processing of the entire image processing unit determined in step 522 has become a value for changing the execution priority of the thread corresponding to each image processing module 38. . Note that it is not necessary to frequently change the execution priority of the thread, and in order to avoid adding an extra load to the CPU 12 by frequently changing the execution priority, as a determination condition in the determination of step 524 For example, the above determination is affirmed every time the progress of image processing increases by 10% since the last change of thread execution priority (or initial setting), so that it does not cause an excessive load. A determination condition in which the thread execution priority is changed at sparse intervals may be used.

  If the determination is negative, the block unit control process 3 is terminated without performing any processing. If the determination is affirmative, in step 526, the center of the execution priority set for each thread at the initial setting is determined. For threads that set a high execution priority at the time of initial setting with reference to the value (or average value), the execution priority gradually decreases as image processing proceeds, and a thread that sets a low execution priority at the time of initial setting For the image processing, the execution priority of the thread corresponding to each image processing module 38 is changed and set so that the execution priority gradually increases with the progress of the image processing, and then the block unit control processing 3 is terminated.

  Note that the change in the execution priority in this step 528 increases the change amount of the execution priority of the corresponding thread as the position of the image processing module 38 approaches the foremost stage or the last stage, and as an example, FIG. As shown in (C), at the end of the image processing of the entire image processing unit, the thread execution priority corresponding to the preceding image processing module 38 and the thread execution priority corresponding to the succeeding image processing module 38 are displayed. The degree relationship may be reversed. For example, as shown in FIGS. 9D and 9E, each image processing module 38 corresponds to each image processing module 38 at the final stage of image processing as a whole. The execution priority of the thread to be executed may be constant. As the series of image processing in the image processing unit progresses, the execution priority of the thread corresponding to each image processing module 38 is changed as described above, so that the CPU 12 can be used effectively and the image can be processed with high processing efficiency. Processing can be performed. Note that the initial setting and changing of the execution priority of the thread corresponding to each image processing module 38 as described above corresponds to the invention of claim 14.

  In addition, while the image processing unit 50 performs image processing, the individual image processing modules 38 and the individual buffer modules 40 (corresponding to the threads) constituting the image processing unit 50 are managed as described above. The acquisition and release of the memory are repeated via the unit 46B. Here, when a thread corresponding to a certain module requests memory acquisition to the resource management unit 46B, but the memory cannot be acquired and an error is returned from the resource management unit 46B, the module performs the subsequent processing. Since the operation cannot be performed normally, the entire processing in the image processing unit 50 ends in failure. For this reason, the resource management unit 46B according to the present embodiment performs the processing described below in order to avoid the failure of memory acquisition from immediately causing the entire processing in the image processing unit 50 to fail.

  That is, the resource management unit 46B performs the memory acquisition request processing shown in FIG. 11 each time a memory acquisition request is input from an arbitrary thread corresponding to an arbitrary module (the image processing module 38 or the buffer module 40). In this memory acquisition request processing, first, in step 600, it is determined whether or not the queue (memory acquisition queue) for registering a thread requesting memory acquisition is empty (the acquisition request information is not registered). . If this determination is affirmative, it can be determined that there is no thread that is requesting memory acquisition other than the thread corresponding to the memory acquisition request input this time. For this reason, when the determination in step 600 is affirmed, the process proceeds to step 602, the operating system 30 is inquired about the maximum size of the memory area (continuous area) that can be secured through the operating system 30, and the maximum size of the memory area that can be acquired To get.

  In the next step 604, the maximum size acquired in step 602 is compared with the request size included in the memory acquisition request input this time, and it is determined whether or not the maximum size is equal to or larger than the request size. It is determined whether or not a memory area having the size requested by the memory acquisition request can be secured. If the determination is affirmative, the process proceeds to step 606, a memory area of the requested size is secured through the operating system 30, and the memory address secured by notifying the start address of the secured memory area to the thread that requested the memory acquisition Is transferred to the memory acquisition request source thread, and the memory acquisition request processing ends. In this case, the module corresponding to the memory acquisition request source thread can normally continue the subsequent processing.

  If the maximum size that can be secured through the operating system 30 is less than the request size from the thread of the memory acquisition request source, the determination in step 604 is denied and the process proceeds to step 612 to determine the memory acquisition request source. By recognizing the thread ID and notifying the recognized ID to the operating system 30, execution of the memory acquisition requesting thread by the CPU 12 is stopped through the operating system 30. In step 614, the execution priority of the memory acquisition request source thread is recognized, and the ID, execution priority, and request size of the memory acquisition request source thread are acquired as acquisition request information together with the registration time in the memory acquisition queue. Register in the memory acquisition queue and end the memory acquisition request processing. In the present embodiment, when there are a plurality of acquisition request information (when there are a plurality of threads requesting memory acquisition), the execution priority of the thread corresponding to each acquisition request information on the memory acquisition queue However, if the determination in step 600 is affirmative as described above, the memory acquisition queue is empty, so the above acquisition request information is stored in the memory acquisition queue. Simply register with.

  Also, when a memory acquisition request is input from an arbitrary thread, if the acquisition request information has already been registered in the memory acquisition queue, the determination in step 600 is denied and the process proceeds to step 608, and this time input The execution priority of the thread corresponding to the issued memory acquisition request is recognized. In the next step 610, the acquisition request information registered at the head of the memory acquisition queue is referred to, the execution priority of the thread corresponding to the acquisition request information is recognized, and the execution priority recognized in step 608 is recognized. It is determined whether or not the execution priority of the thread corresponding to the memory acquisition request input this time is higher than the execution priority of the thread corresponding to the acquisition request information registered at the head of the memory acquisition queue. If the determination is negative, the process proceeds to step 612, and as described above, the execution stop of the thread of the memory acquisition request source (step 612) and the acquisition request information registration to the memory acquisition queue (step 614) are sequentially performed.

  If the determination in step 610 is affirmative, the process proceeds to step 602 to acquire the maximum size of the memory area that can be secured (step 602) and whether or not the memory area requested by the current memory acquisition request can be secured. These determinations (step 604) are performed in order. If the requested memory area can be secured, the memory area is secured and transferred to the acquisition request source thread (step 606). If the requested memory area cannot be secured, execution of the thread is stopped. The acquisition request information is registered in the memory acquisition queue (steps 612 and 614).

  As described above, when a memory acquisition request is input but a memory area of the requested size cannot be allocated to the acquisition request source thread, the operation of the acquisition request source thread is stopped and the acquisition request information is sent to the memory acquisition queue. During this time, processing is stopped even in the module corresponding to the thread whose operation is stopped, but it is avoided that a failure to acquire a memory area in an arbitrary thread immediately causes a failure of the entire processing in the image processing unit 50. be able to.

  Further, each time the memory release request is input from an arbitrary thread corresponding to an arbitrary module (the image processing module 38 or the buffer module 40), the resource management unit 46B performs the memory release request processing shown in FIG. In this memory release request processing, first, in step 620, the memory area requested to be released is released through the operating system 30. In the next step 622, it is determined whether or not the memory acquisition queue is empty (the acquisition request information is not registered). If this determination is affirmative, the memory release request processing is terminated without performing any processing.

  If the acquisition request information is registered in the memory acquisition queue, the determination in step 622 is denied and the process proceeds to step 624, where the maximum size of the memory area (continuous area) that can be secured through the operating system 30 is determined. To the operating system 30 to obtain the maximum size of the memory area that can be obtained. In the next step 626, the maximum size acquired in step 624 is compared with the request size included in the acquisition request information registered at the top of the memory acquisition queue to determine whether the maximum size is equal to or larger than the request size. Thus, it is determined whether or not a memory area having the size requested by the acquisition request information registered at the top of the memory acquisition queue can be secured. If this determination is negative, the memory release request processing ends without performing any processing. In this case, the thread corresponding to the acquisition request information registered in the memory acquisition queue is maintained in a stopped state.

  If the determination in step 626 is affirmative (if it is determined that a memory area having the size requested by the acquisition request information registered at the top of the memory acquisition queue can be secured), the process proceeds to step 628. The acquisition request information is extracted from the head of the memory acquisition queue, the ID included in the acquired acquisition request information is notified to the operating system 30, and the execution stop state of the thread corresponding to the acquired acquisition request information is transmitted through the operating system 30. Release it. In step 630, a memory area of the requested size included in the retrieved acquisition request information is secured through the operating system 30, and the start address of the secured memory area is notified to the thread that has released the execution suspended state in step 628. The secured memory area is handed over to the thread that requested the memory acquisition from which the execution stop state was released. As a result, the module corresponding to the thread whose execution stop state has been released resumes the stopped processing and can normally perform the subsequent processing by acquiring the requested memory area.

  When the process of step 630 ends, the process returns to step 622, and steps 622 to 630 are repeated until the determination of step 622 is affirmed or the determination of step 626 is negative. As a result, even when multiple pieces of acquisition request information are registered in the memory acquisition queue, whether or not the memory area requested for acquisition by each acquisition request information can be secured in order (execution of the corresponding thread). If it can be determined and secured, the execution stop state of the corresponding thread is canceled (step 628), and the memory area is secured and delivered (step 630).

  As described above, in this embodiment, when the memory area requested to be acquired is not assignable, the operation of the acquisition request source thread is stopped without outputting an error, thereby corresponding to the thread. When the memory area requested for acquisition is interrupted, the operation of the suspended thread is resumed, and the memory area requested for acquisition is allocated and allocated. Even if the memory area is temporarily insufficient, the entire processing in the image processing unit 50 can be continued. In addition, when there are a plurality of threads requesting acquisition, it is determined whether or not the requested memory area can be allocated (and allocation of the memory area when allocation is possible). Since the requested memory area cannot be secured and the execution of multiple threads is stopped, the memory area that can be secured is increased and the thread is released from the suspended execution state and the memory area is allocated. Even in this case, the subsequent processing in the image processing unit 50 can be performed with high processing efficiency.

  Further, when the memory area requested for acquisition cannot be secured, for example, depending on the usage status of the memory by other application programs executed in parallel with a plurality of threads corresponding to the image processing unit 50 by the CPU 12, the acquisition may be performed. Since it takes a very long time until the required memory can be secured, the waiting time of the user waiting for the completion of the processing in the image processing unit 50 becomes extremely long, and a great burden is placed on the user. There is a possibility of hanging. Considering the above, the resource management unit 46B according to the present embodiment performs the waiting time monitoring process shown in FIG. 13 at regular time intervals.

  In this waiting time monitoring process, it is first determined in step 640 whether or not the memory acquisition queue is empty. If the determination is affirmative, the waiting time monitoring process is terminated without performing any process. If acquisition request information is registered in the memory acquisition queue, the process proceeds from step 640 to step 642 and 1 is assigned to the counter n. In the next step 644, the nth acquisition request information registered in the memory acquisition queue is referred to, and the registration time included in the acquisition request information (the registration time of the acquisition request information in the memory acquisition queue) is recognized. To do. In the next step 646, it is determined whether or not the elapsed time from the registration time recognized in step 644 is larger than a preset threshold value.

  If the determination is negative, the process proceeds to step 648 and the value of the counter n is incremented by 1. In the next step 650, it is determined whether or not the nth acquisition request information is registered in the memory acquisition queue. If the determination is affirmative, the process returns to step 644, and steps 644 to 650 are repeated until the determination at step 646 is affirmed or the determination at step 650 is negative. In Steps 644 to 650, it is searched whether or not the acquisition request information registered in the memory acquisition queue includes acquisition request information whose elapsed time from the registration time exceeds the threshold. If there is no corresponding acquisition request information, the determination in step 650 is denied, and the waiting time monitoring process is terminated without performing any process.

  If the acquisition request information registered in the memory acquisition queue includes acquisition request information whose elapsed time from the registration time exceeds the threshold, the determination in step 646 is affirmed and the step The process moves to 652, where all the acquisition request information registered in the memory acquisition queue is extracted, and the ID corresponding to the acquired acquisition request information is notified to the operating system 30, whereby the thread corresponding to the acquired acquisition request information Is stopped by the operating system 30. In step 654, an error response to the memory acquisition request is returned to all threads whose execution stopped state has been canceled. As a result, when an error occurs in the processing corresponding to each thread, the entire image processing in the image processing unit 50 is stopped, and the user waiting for completion of the processing in the image processing unit 50 is stopped. Since the waiting time becomes extremely long, it is possible to prevent a great burden on the user.

  In the above, as a memory management method by the resource management unit 46B, a management method for securing a memory area to be allocated from the memory 14 to the request source module through the operating system 30 for each request from each module of the image processing unit 50. Although described in the premise, the present invention is not limited to this, and a memory area of a certain size is secured in advance through the operating system 30 from the memory 14, and if there is a request from each module, the size of the requested memory area If the size of the requested memory area is equal to or greater than the threshold, the operating system allocates a memory area to be allocated to the request source module. The present invention is suitable for a management system secured through It is also possible to.

  In addition, in the above, every time a memory release request is input from an arbitrary thread, whether the memory area requested for acquisition can be secured by the acquisition request information registered in the memory acquisition queue by the memory release request processing Although the aspect (equivalent to the invention described in claim 4) has been described, the present invention is not limited to this, and it is determined at regular intervals whether or not the memory area can be secured. (It is equivalent to the invention of claim 3).

  In addition, in the above, if there is acquisition request information in the memory acquisition queue that has exceeded the threshold for the elapsed time since registration in the memory acquisition queue, an error response is sent to the memory acquisition request source thread. Returning, the aspect (corresponding to the invention described in claim 6) of stopping the entire image processing in the image processing unit 50 has been described, but when a memory acquisition request is input from a thread corresponding to an arbitrary image processing module 38, Threads corresponding to all other image processing modules 38 are in a state where acquisition request information is registered in the memory acquisition queue, or waiting until image data can be acquired from the previous module (image processing It is determined whether or not the module control process (steps 222 and 224 of FIG. 6 is repeated), and even if the above condition is met, a memory acquisition request It may be caused to stop the entire image processing in the image processing unit 50 thread returns an error response (corresponding to the invention of claim 7, wherein).

  Furthermore, in the above, when a memory release request is input from an arbitrary thread, it is possible to determine whether or not the requested memory area can be secured by the acquisition request information registered in the memory acquisition queue. Although there has been described a mode (corresponding to the invention according to claim 10) in which the process of allocating the memory area is performed in descending order of the execution priority of the thread corresponding to each acquisition request information, the present invention is limited to the above mode Instead of registering acquisition request information in the memory acquisition queue in the order of input of the memory acquisition request, when a memory release request is input from any thread, the acquisition request information is extracted from the top of the memory acquisition queue, It is determined whether or not the memory area requested by the acquired acquisition request information can be secured. The acquired acquisition request information may be re-registered at the end of the memory acquisition queue (the invention according to claim 9 and claim 11 “the allocation of storage resources was requested from the first execution unit program” Sometimes, when an allocation request from the second execution unit program is already registered in the queue, the allocation request from the first execution unit program is registered in the queue as it is). In this case, although the execution priority of the thread corresponding to each acquisition request information is not considered, the execution stop time of the thread corresponding to each acquisition request information (waiting until the memory is acquired after the memory acquisition request is output) Time) can be averaged.

  In the above, when a memory acquisition request is input from an arbitrary thread, the execution priority of the thread that is the memory acquisition request source is the execution of the thread corresponding to the acquisition request information registered at the top of the memory acquisition queue. It is determined whether or not the priority is higher than the priority, and if the determination is negative, the acquisition request information is registered in the memory acquisition queue without determining whether or not the requested memory area can be secured (request) However, the present invention is not limited to this, and when a memory acquisition request is input from an arbitrary thread, it is determined whether or not the requested memory area can be secured. If the allocation is possible, the requested memory area may be allocated to the memory acquisition request source thread, and if the allocation is not possible, the acquisition request information may be registered in the memory acquisition queue. It corresponds to the invention of claim 11).

  Further, in the above, the execution priority of each thread corresponding to each image processing module 38 becomes higher as the position of the image processing module 38 in the connected form in the pipeline form or the directed acyclic graph form becomes higher. As described above, the aspect (corresponding to the invention of claim 14) has been described in which the initial setting is made and the image processing is changed in accordance with the degree of progress of the image processing. However, the present invention is not limited to this. When the image processing in the module 38 is image processing with a large variation in the load applied to the CPU 12, the image processing module 38 receives the image processing module 38 according to the result of estimating the processing load of the individual image processing module 38 in advance. The execution priority of the corresponding thread may be fixedly set (corresponding to the invention of claim 13).

  In addition, the execution priority of the thread corresponding to each image processing module 38 is set to the image processing module 38 (position value = position of the own module) connected to the subsequent stage via the subsequent buffer module 40 for each image processing module. Although the read request is input from the image processing module 38) of the value +1 to the subsequent buffer module 40, the effective data stored in the subsequent buffer module 40 is less than the unit read data amount. The number of times (wait) that a data request is input from the buffer module 40 and “wait” (waiting until valid data in the buffer module 40 exceeds the unit read data amount) is generated in the subsequent image processing module. It may be changed according to the number of occurrences).

  Specifically, the workflow management unit 46A holds the number of waiting occurrences (initial value = 0) for each image processing module 38 in advance, and every time a data request is input from an arbitrary buffer module 40, the data The number of waiting occurrences is incremented by 1 for the image processing module 38 preceding the request input source buffer module 40. Then, the workflow management unit 46A calculates an average value of the number of waiting occurrences held for each of the image processing modules 38 at a constant time period, and calculates the average value of the calculated waiting occurrence times and the individual image processing modules. The execution priority of the thread corresponding to each image processing module 38 is changed according to the deviation of the number of occurrences of waiting 38. For the image processing module 38 in which the number of waiting occurrences is greater than the average value, the change in the execution priority increases as the execution priority of the corresponding thread increases, and the number of waiting occurrences exceeds the average value. For a small number of image processing modules 38, the execution priority of the corresponding thread can be reduced as the deviation increases. Specifically, for example, the following priority can be given.

Execution priority change rate (%) = (Number of wait occurrences-Average value of wait occurrences) / Average value x 100
Execution priority after change = execution priority + (execution priority x change rate) / 100
In the calculation, the median value of the number of waiting occurrences may be used instead of the average value of the number of waiting occurrences.

  The image processing module 38 in which the number of waiting occurrences is greater than the average value causes the image processing module 38 connected to the subsequent stage via the subsequent buffer module 40 to generate a relatively large number of “waits”. Although it can be determined that the image processing in the module 38 is a bottleneck of the image processing of the entire image processing unit, the execution priority of the thread corresponding to the image processing module 38 is increased in the above processing. In addition, the image processing module 38 in which the number of occurrences of waiting is smaller than the average value has a relatively small number of occurrences of “waiting” in the image processing module 38 connected to the subsequent stage via the buffer module 40 in the subsequent stage. Prioritizing image processing in another image processing module 38 that has a relatively high number of occurrences of waiting than the module 38 can improve the efficiency of the image processing of the entire image processing unit. The execution priority of the thread corresponding to is reduced.

  Thereby, the execution priority of the thread corresponding to each image processing module 38 can be optimized according to the number of waiting occurrences in the subsequent image processing module 38 (and the deviation from the average value of the number of waiting occurrences). The CPU 12 can be used effectively to perform image processing with high processing efficiency. Also, the acquisition request information registered in the memory acquisition queue is arranged according to the execution priority set as described above, so that the execution of multiple threads can be stopped without securing the requested memory area. Even when the memory area can be allocated after the execution stop state of the thread is released due to an increase in the memory area that can be secured, the subsequent processing in the image processing unit 50 can be performed with high processing efficiency. Note that setting / changing the execution priority of the thread corresponding to each image processing module 38 as described above corresponds to the invention of claim 16.

  In the above aspect, the number of times a data request is input from the subsequent buffer module 40, that is, the number of times “waiting” has occurred in the image processing module 38 connected to the subsequent stage via the subsequent buffer module 40, The image processing module 38 has written the image data to the subsequent buffer module 40, and waits for the number of times that the effective data of the subsequent buffer module 40 has not reached the unit read data amount of the subsequent image processing module 38. You may make it use as a frequency | count. In this case, the number of waiting occurrences is preferable because it more accurately reflects the degree of “waiting” in the subsequent image processing module 38.

  In the above aspect, the execution priority of the thread corresponding to each image processing module 38 is changed according to the number of occurrences of “waiting” in the subsequent image processing module 38 connected via the subsequent buffer module 40. In addition, the execution priority of the thread is changed according to the number of occurrences of “wait” in the own module (specifically, the thread corresponding to the image processing module 38 having a relatively large number of occurrences of waiting has its execution priority set). The thread corresponding to the image processing module 38 having a relatively small number of occurrences of waiting may be increased in execution priority).

  In addition, the execution priority of the thread corresponding to each image processing module 38 is set to the unit data amount when the image processing module 38 subsequent to each buffer module 40 acquires image data from each buffer module 38. You may make it change according to the ratio of the data amount of the image data memorize | stored in each buffer module 40. FIG. This can be realized, for example, by the workflow management unit 46A performing the following processing at a constant time period.

  That is, first, the current accumulated data amount of each buffer module 40 is acquired, and the ratio of the current accumulated data amount of each buffer module 40 to the unit read data amount of the image processing module 38 at the subsequent stage is obtained for each buffer module 40. Calculate. For example, the accumulated data amount is a value represented by the number of lines of the image, the accumulated data amount in a certain buffer module 40 is “10 lines”, and the unit read data amount of the image processing module 38 in the subsequent stage of the buffer module 40 is “ If it is “1 line”, the ratio of the accumulated data amount is 10/1 = 10, and if the unit read data amount of the image processing module 38 in the subsequent stage is “8 lines”, the ratio of the accumulated data amount is 10/8 = 1.25. Then, the average value of the ratio of the accumulated data amount calculated for each buffer module 40 is calculated, and the average value of the calculated ratio of the accumulated data amount and the deviation of the ratio of the accumulated data amount of each buffer module 40 are calculated. The execution priority of the thread corresponding to each image processing module 38 in the previous stage of each buffer module 40 is changed.

  This change in the execution priority of the image processing module 38 in the preceding stage of the buffer module 40 in which the ratio of the accumulated data amount is smaller than the average value increases as the execution priority of the corresponding thread increases. For the preceding image processing module 38 of the buffer module in which the ratio of the accumulated data amount is smaller than the average value, the execution priority of the corresponding thread can be increased as the deviation increases. Specifically, it can be performed according to the following formula, for example.

Execution priority change rate (%) = (average value-ratio of accumulated data) / average value x 100
Execution priority after change = original execution priority + (execution priority x change rate) / 100
In the above calculation, the median value of the ratio of the accumulated data amount may be used instead of the average value of the ratio of the accumulated data amount.

  The buffer module 40 in which the ratio of the accumulated data amount is smaller than the average value has a smaller amount of effective data than the unit read data amount in the subsequent image processing module 38, and is relatively many times in the subsequent image processing module 38. The “waiting” is likely to have occurred, and the image processing in the image processing module 38 in the preceding stage of the buffer module 40 is highly likely to be a bottleneck of image processing for the entire image processing unit. In the above process, the execution priority of the thread corresponding to such an image processing module 38 is increased. Further, since the buffer module 40 in which the ratio of the accumulated data amount is larger than the average value stores valid data having a sufficient data amount as compared with the unit read data amount in the subsequent image processing module 38, the buffer Prioritizing the image processing in the image processing module 38 in the preceding stage of the other buffer module 40 having a relatively small ratio of the accumulated data amount than the image processing module 38 in the preceding stage of the module performs image processing as the entire image processing unit. Although the efficiency can be improved, in the above process, the execution priority of the thread corresponding to the image processing module 38 is lowered.

  As a result, the execution priority of the thread corresponding to each image processing module 38 is optimized according to the ratio of the accumulated data amount in the subsequent buffer module 40 (and the deviation from the average value of the accumulated data amount ratio). As a result, the CPU 12 can be used effectively to perform image processing with high processing efficiency. Also, the acquisition request information registered in the memory acquisition queue is arranged according to the execution priority set as described above, so that the execution of multiple threads can be stopped without securing the requested memory area. Even when the memory area can be allocated after the execution stop state of the thread is released due to an increase in the memory area that can be secured, the subsequent processing in the image processing unit 50 can be performed with high processing efficiency. Note that setting / changing the execution priority of the thread corresponding to each image processing module 38 as described above corresponds to the invention described in claim 15.

  In the above description, the memory 14 is described as an example of the storage resource according to the present invention. However, the present invention is not limited to this, and is connected to the computer 10 via an HDD, a flash memory, or a communication line in addition to the memory. It is also possible to apply various external storage devices as storage resources. In addition, when a storage device to be secured as a storage resource is configured to be selectable from a plurality of types of storage devices as described above, the access speed to the storage resource depends on the type of storage device used as the storage resource, and the storage device Since there are multiple threads that request acquisition of storage resources, a storage resource with a high access speed is assigned to a thread with a high execution priority. It is preferable to preferentially assign to.

  Furthermore, in the above description, the mode in which the individual modules (programs) constituting the image processing unit 50 are executed as separate threads when the image processing unit 50 is operated in the parallel processing method has been described. However, when the image processing module 38 reads image data from the preceding buffer module 40 and writes image data or the like to the subsequent buffer module 40, image processing or the like in the image processing module 38 is temporarily performed. In consideration of the stopped state, the program corresponding to the buffer control unit of each buffer module 40 is replaced with the program of the write control unit 40C (see FIG. 3B) that performs the data writing process, The image processing module is separated into a program of a reading control unit 40D (see FIG. 3B) that performs data reading processing. 38, the program of the read control unit of the buffer module 40 in the preceding stage of the image processing module 38, and the program of the write control unit of the buffer module 40 in the subsequent stage of the image processing module 38 are executed as a single thread. Alternatively, the above-described thread may be generated for each image processing module 38 constituting the image processing unit 50 and executed in parallel to cause the image processing unit 50 to operate in a parallel processing method.

It is a block diagram which shows schematic structure of the computer (image processing apparatus) which concerns on this embodiment. It is a block diagram which shows the structural example of an image process part. (A) is an image processing module, and (B) is a block diagram showing a schematic configuration of a buffer module and processing to be executed. It is the schematic explaining the case where the image data to be written straddles a plurality of storage unit buffer areas. It is the schematic explaining the case where the image data to be read straddles a plurality of storage unit buffer areas. It is a flowchart which shows the content of the image processing module control process performed by the control part of an image processing module. It is a flowchart which shows the content of the block unit control process performed by the process management part. It is the schematic explaining the flow of the image processing in an image process part. It is the schematic which shows an example of transition of the execution priority of the thread | sled corresponding to each image processing module with progress of a series of image processing in an image process part. It is a block diagram for demonstrating the definition of the position of the image processing module in the connection form of a pipeline form or a directed acyclic graph form. It is a flowchart which shows the content of the memory acquisition request | requirement process performed by the resource management part. It is a flowchart which shows the content of the memory release request | requirement process performed by the resource management part. It is a flowchart which shows the content of the waiting time monitoring process performed by the resource management part.

Explanation of symbols

10 Computer 12 CPU
14 memory 20 storage unit 22 image data supply unit 24 image output unit 38 image processing module 38A image processing engine 38B control unit 40 buffer module 46 processing management unit 46A workflow management unit 46B resource management unit 50 image processing unit

Claims (17)

Processing for acquiring image data from the previous stage of the own module, performing predetermined image processing on the acquired image data, and outputting the image data that has undergone the image processing or the processing result of the image processing to the subsequent stage of the own module A plurality of image processing modules having a function to be used are connected in a pipeline form or a directed acyclic graph form, and correspond to and correspond to different one or a plurality of image processing modules. A plurality of execution unit programs that request allocation and release of storage resources used by the image processing unit that are executed in parallel by the program execution resources; and
Each time a storage resource allocation is requested from an arbitrary execution unit program, it is determined whether or not the storage resource requested for allocation can be allocated to the execution unit program of the allocation request source. The operation of the execution unit program of the allocation request source is stopped, and when it is determined that the storage resource requested to be allocated can be allocated to the execution unit program of the allocation request source that has stopped the operation, Management means for allocating the storage resource requested for allocation to the execution unit program of the allocation request source that has resumed the operation, resuming the operation of the execution request program of the allocation request source that has stopped the operation;
An image processing apparatus.   The image processing unit includes a buffer composed of storage resources in at least one of the preceding stage and the succeeding stage of each of the plurality of image processing modules, and when the image processing module is connected to the preceding stage of its own module, The image data output from the image processing module is written in the buffer, and when the image processing module is connected to the subsequent stage of the own module, the image data stored in the buffer is converted to the subsequent image processing module. The execution unit program corresponds to at least one of different single or plural image processing modules and different single or plural buffer modules, and corresponds to the buffer module. The execution unit program has a corresponding buffer The image processing apparatus according to claim 1, wherein the requesting allocation and release of memory resources used by the module.   The management means determines the storage resource requested to be allocated while there is an execution unit program of the allocation request source that has stopped the operation by determining that the storage resource requested to be allocated cannot be allocated. The image processing apparatus according to claim 1, wherein the image processing apparatus periodically determines whether the assignment request source execution unit program can be assigned.   The management means follows a request from another execution unit program while there is an execution unit program of the allocation request source whose operation is stopped by determining that the storage resource requested to be allocated cannot be allocated. 3. The method according to claim 1, wherein each time a storage resource is released, it is determined whether or not the storage resource requested to be allocated can be allocated to the execution unit program of the allocation request source. Image processing apparatus. Each said execution unit program notifies the amount of storage resources required at the time of a storage resource allocation request,
Whether the management means can allocate the storage resource requested for allocation to the execution unit program of the allocation request source by determining whether or not the allocated storage resource amount exceeds the notified required storage resource amount. The image processing apparatus according to claim 1, wherein a determination is made as to whether or not.   The management means outputs an error when the operation stop time of the allocation request source execution unit program whose operation is stopped by determining that the storage resource requested to be allocated cannot be allocated exceeds a predetermined time. The image processing apparatus according to claim 1, wherein the processing by the image processing unit is stopped.   The management unit determines that the storage resource requested to be allocated from an arbitrary execution unit program cannot be allocated, and all other execution unit programs corresponding to the image processing unit store the storage requested for allocation. The first state in which the operation is stopped by determining that the resource cannot be allocated, or the second state in which the corresponding image processing module cannot acquire image data from the previous stage of its own module and stops the processing. The image processing apparatus according to claim 1, wherein an error is output and processing by the image processing unit is stopped when the state is in a state.   The management means registers the allocation request from the execution unit program of the allocation request source whose operation is stopped by determining that the storage resource requested to be allocated cannot be allocated to the queue, and at an arbitrary timing The allocation request registered in the queue is extracted, and whether or not the storage resource requested for allocation by the extracted allocation request can be allocated to the execution unit program of the allocation request source corresponding to the extracted allocation request. The image processing apparatus according to claim 1, wherein the determination is performed.   The management means retrieves an allocation request from the queue, and determines whether or not the storage resource requested for allocation can be allocated by the retrieved allocation request. As a result, the storage resource is not allocated. And when the allocation request is registered again in the queue, and when a plurality of allocation requests are registered in the queue, a single allocation request is extracted from the queue and extracted. The process of determining whether or not the storage resource requested to be allocated can be allocated by a single allocation request is performed in the order of registration in the queue with respect to the plurality of allocation requests. The image processing apparatus according to claim 8.   The management means retrieves a single allocation request from the queue when a plurality of allocation requests are registered in the queue, and stores a storage resource whose allocation is requested by the single allocation request that has been extracted. The processing for determining whether or not allocation is possible is performed in descending order of execution priority of an execution unit program of an allocation request source corresponding to each allocation request registered in the queue. 8. The image processing apparatus according to 8.   When the allocation request from the second execution unit program is already registered in the queue when the allocation of storage resources is requested from the first execution unit program, the management means The allocation request from the execution unit program is registered in the queue as it is, or it is determined whether the storage resource requested for allocation by the allocation request from the first execution unit program can be allocated, and the storage When it is determined that the resource can be allocated, the storage resource requested to be allocated is allocated to the first execution unit program, and when it is determined that the storage resource cannot be allocated, the allocation from the first execution unit program is performed. The image processing apparatus according to claim 8, wherein a request is registered in the queue.   The management means executes the first execution when the allocation request from the second execution unit program is already registered in the queue when the storage resource allocation is requested from the first execution unit program. It is determined whether the execution priority of the unit program is higher than the execution priority of the second execution unit program, and the execution priority of the first execution unit program is higher than the execution priority of the second execution unit program Sometimes, it is determined whether or not the storage resource requested for allocation can be allocated by an allocation request from the first execution unit program. If the storage resource can be allocated, the allocation to the first execution unit program is performed. Allocates the requested storage resource, and if the storage resource cannot be allocated, registers the allocation request from the first execution unit program in the queue On the other hand, when the execution priority of the first execution unit program is less than or equal to the execution priority of the second execution unit program, the allocation request from the first execution unit program is registered in the queue as it is. The image processing apparatus according to any one of claims 8 to 10.   The said management means sets the execution priority of each corresponding execution unit program according to the result which estimated the processing load of each image processing module which comprises the said image processing part previously. The image processing apparatus according to claim 10 or 12.   The management means determines the execution priority of the corresponding individual execution unit program according to the position of the individual image processing modules constituting the image processing unit in the pipeline form or the connected form of the directed acyclic graph form. 13. The image processing apparatus according to claim 10, wherein initial setting is performed and the execution priority of each execution unit program is changed according to the degree of progress of image processing in the image processing unit. The image processing unit includes a buffer composed of storage resources in at least one of the preceding stage and the succeeding stage of each of the plurality of image processing modules, and when the image processing module is connected to the preceding stage of its own module, The image data output from the image processing module is written in the buffer, and when the image processing module is connected to the subsequent stage of the own module, the image data stored in the buffer is converted to the subsequent image processing module. The execution unit program corresponds to at least one of different single or plural image processing modules and different single or plural buffer modules, and corresponds to the buffer module. The execution unit program has a corresponding buffer Requesting allocation and release of memory resources used by the module,
The management means is stored in the individual buffer module as the degree of progress of the image processing with respect to a unit data amount when an image processing module subsequent to the individual buffer module acquires image data from the individual buffer module. 15. The image processing according to claim 14, wherein the execution priority of each corresponding execution unit program is changed in accordance with the ratio in the individual buffer modules using a ratio of the amount of image data being stored. apparatus.   The management means uses the number of image data acquisition failures in each image processing module as the degree of progress of the image processing, and the corresponding individual execution unit program according to the number of acquisition failures in the individual image processing modules The image processing apparatus according to claim 14, wherein the execution priority of is changed. A computer having storage resources and program execution resources;
Processing for acquiring image data from the previous stage of the own module, performing predetermined image processing on the acquired image data, and outputting the image data that has undergone the image processing or the processing result of the image processing to the subsequent stage of the own module A plurality of image processing modules having a function to be used are connected in a pipeline form or a directed acyclic graph form, and correspond to and correspond to different one or a plurality of image processing modules. An image processing unit that operates by executing a plurality of execution unit programs that request allocation and release of storage resources used by the program execution resources in parallel;
Each time storage resource allocation is requested by an arbitrary execution unit program, it is determined whether or not the storage resource requested for allocation can be allocated to the execution unit program of the allocation request source, and it is determined that allocation is not possible. When it is determined that the operation of the execution unit program of the allocation request source is stopped and the storage resource requested to be allocated can be allocated to the execution unit program of the allocation request source that has stopped the operation. Management means for resuming operation of the execution unit program of the allocation request source that has stopped the operation and allocating the storage resource requested for allocation to the execution unit program of the allocation request source that has resumed the operation An image processing program to function.

Images