JP5906157B2 - Image processing device - Google Patents

Description

  The present invention relates to a plurality of image processing operations including continuous processing steps, and more particularly to image recognition that requires real-time characteristics.

  In recent years, in-vehicle image recognition systems that recognize surrounding vehicles or traveling lanes through cameras and prevent accidents are becoming widespread. The objects to be recognized are diversified, and it is necessary to recognize pedestrians, road signs, and the like outside the vehicle. For this purpose, the image recognition system includes a plurality of detection algorithms. In addition, since it is necessary to instantly grasp the surrounding situation, processing time is severely limited, and it is necessary to shorten the processing time of the image recognition system as much as possible so that it can operate in real time.

  When the detection algorithms of a plurality of detection algorithms are performing exactly the same processing steps, it is desirable to execute the processing steps only once in order to reduce processing time. Therefore, it is necessary to overlook and integrate a plurality of detection algorithms. However, there is a problem that man-hours are required to extract common processing steps and redesign the algorithm.

  Patent Document 1 describes a technique for shortening the processing time so that processing steps that have already been performed can be omitted in the subsequent processing regarding the processing method of X-ray diffraction measurement data. Here, when there is an instruction to use the processing data 1, 2, 3, the processing data 1, 2, 3 stored in the processing A file is used as the processing data of the processing steps 1, 2, 3 in the processing program B. It is described to be used as.

  Patent Document 2 addresses each processing data obtained by the execution of each processing step when the processing consisting of an arbitrary continuous processing step is executed on the measurement data with respect to the same problem that occurs in the X-ray analysis. When the processing data is stored in a file in association with the processing step and the processing data is the same as a plurality of continuous processing steps from the first processing step in which the processing data is stored, the processing redundancy is specified by specifying the processing data storage file. Is omitted. Since this mechanism is incorporated in the processing process, the use of this mechanism does not require man-hours for extracting a common processing process and redesigning the algorithm.

JP-A-11-64250 Japanese Patent No. 3739021

  In Patent Document 1, a processing file designated by a user is read out, and processing data relating to the same processing step is extracted from the processing file. There is a problem that it is necessary every time the program is executed.

  In Patent Literature 2, when a process that has already been executed is to be reused, the user needs to specify an instruction to use process data. In-vehicle image recognition processing that does not require user judgment, it is necessary to automate judgment at the time of execution. Further, since the processing result is stored every time, the overhead required for real-time processing becomes a problem.

  On the other hand, with respect to the former problem, it is considered that a common processing step can be found by analyzing the program in advance due to recent advances in program analysis technology, etc. I do not. An object of the present invention is to reduce overhead associated with eliminating redundancy of processing steps during processing.

  In order to solve the above-described problems, an image processing apparatus according to the present invention is an image processing apparatus including a plurality of image processes including continuous processing steps, and the same processing step is provided between the plurality of image processes. In an image processing apparatus that reuses output image data of a first processing step for the same processing step after the second time, an image information management unit that holds information on whether or not to reuse each image; And overwriting only when the image to be overwritten by the processing step based on the information is an image to be reused and an image to be reused. Image duplicating means for creating a duplication of the image data in advance.

  The image processing method according to the present invention is an image processing method composed of a plurality of image processes composed of continuous processing steps, and when there is an identical processing step between the plurality of image processing, the first processing step In the image processing method for reusing the output image data for the second and subsequent identical processing steps, a step of holding information on whether or not to reuse each image, and a processing step using the information Determining whether an image to be overwritten is an image to be reused, and creating a copy of the image data prior to overwriting only when the image is to be reused It is characterized by including these.

  According to the present invention, the redundant processing can be basically automated by reducing the user's work, the integration man-hours of the algorithm can be reduced, and the processing time overhead associated with the automation can be minimized. There is an effect that it can be reduced.

The operation | movement flow of the vehicle-mounted image processing system in the state in which the common processing process was integrated is shown. The operation | movement flow of the vehicle-mounted image processing system which is disperse | distributed for every detection algorithm and is in the state where the common processing process is not integrated is shown. 1 is a conceptual diagram of a configuration of an in-vehicle image processing system according to an embodiment of the present invention. The operation | movement flow of the vehicle-mounted image processing system which is one Example of this invention is shown. 4 shows an example of initial registration contents of an image operation means according to an embodiment of the present invention. As an algorithm according to an embodiment of the present invention, an example including two detection algorithms will be described. An example of initial registration contents of the image operation means when the intermediate value is provided as an intermediate number (801) in the image information management means is shown. The updated registration content of the image operation means immediately after the processing step number (F01) is completed is shown. The updated registration content of the image operation means immediately after the processing step number (F02) is completed is shown. The updated registration content of the image operation means immediately after the processing step number (F03) is completed is shown. The updated registration content of the image operation means immediately after the processing step number (F04) is completed is shown.

[Example]
Embodiments of the present invention will be described in detail below with reference to the drawings. First, a general in-vehicle image processing system and its operation flow will be described.

  FIG. 1 shows an operation flow of an in-vehicle image processing system in a state where common processing steps are integrated. Here, the vehicle-mounted image processing system has a plurality of detection objects, and has a detection algorithm for each detection object. Each detection algorithm consists of a plurality of consecutive processing steps. Examples of detection objects include vehicles, pedestrians, lanes (white lines), signs, and the like. A, B, and C described in FIG. 1 mean different detection objects.

  As the processing steps described in FIG. 1, for example, filter processing and identification processing can be considered. In any of the detection algorithms, a plurality of processes continued from the first image input are common processes. For example, as processing after image input, geometric transformation accompanying lens distortion correction, smoothing processing using a low-pass filter, edge extraction processing using a high-pass filter, or the like can be considered.

  Originally, the flow that operates in the image processing system should be in a state where common processes are integrated as shown in FIG. Here, the algorithm (103) for detecting A and the algorithm (104) for detecting B share processing step 1 (101) and processing step 2 (102), and the algorithm (103) and C for detecting A are shared. The detecting algorithm (105) shares processing step 1 (101).

  However, there are cases where development is performed for each detection algorithm, or already developed algorithms are used. Therefore, even if there are common processing steps, the processing is often distributed for each detection algorithm as in the operation flow shown in FIG. Since the common processing parts (201) and (202) are redundant, the processing time of the operation flow of FIG. 2 is increased compared to the operation flow of FIG.

  Therefore, in order to achieve the processing time of the operation flow of FIG. 1 while maintaining the framework as shown in FIG. 2, the result image of the processing step is temporarily copied and saved, and the same processing step enters the result. A method of reusing images (Patent Document 2) is conceivable, but the overhead for copying is large. Therefore, it is desired to minimize duplication, but to do so, a mechanism is required that duplicates only the results of common processing steps and does not cause duplication when duplication is not necessary.

  Next, terms are defined. Processing steps common to a plurality of detection algorithms, such as processing step 1 and processing step 2 in FIG. 2, are called “shared target processing steps”, and their relationship is called a “pair”. Note that what is actually shared is the image output result of the processing step. The output image of this sharing target processing step is referred to as “sharing target image”. For example, the image processing result of processing step 1 is desired to be used for any of the detection algorithms A to C. If the detection algorithm A first executes the processing step 1, the detection algorithms B and C may use the processing results.

  A unique number is assigned to each processing step, image, and pair, which are referred to as “processing step number”, “image number”, and “pair number”, respectively, and a positive number is entered. However, the processing step number does not indicate the type of the processing step, but a unique number is assigned to each of the entire image processing system.

  Hereinafter, the processing process is simply a process of inputting one image and outputting one image. However, the same mechanism can be used when two or more images are input.

  FIG. 3 is a conceptual diagram of a configuration of an image processing system according to an embodiment of the present invention. In this embodiment, as a framework for minimizing duplication or reuse, an image recognition calculation means (701) for calculating an image and detecting / recognizing / measuring a target object, and a storage medium (311) for managing an image memory and the like And an image operation means (310) for managing image information, common processing steps, and the like, and a process execution adjustment means (702) responsible for optimization of image duplication or reuse.

  When the image recognition calculation means (701) executes the processing step, first, the overwriting image determination means (703) detects whether or not the processing step overwrites the image to be shared. When overwriting occurs, it is determined whether or not the original image is to be copied by the copy necessity determination means (704). The image processing execution management means (705) determines whether to reuse the processed image or newly execute the processing, and actually performs the processing. The determination in each means is performed by referring to the data registered in the image information management means (401), the processing step information management means (402), and the pair information management means (403) in the image operation means (310). Is called. Management of image duplication or reuse is performed by an image duplicating means (404).

  FIG. 4 shows an operation flow of the image duplicating means in which the overwriting image judging means (703), duplication necessity judging means (704) and image processing execution managing means (705) according to the above embodiment are incorporated in each processing step. Show.

  In light of the framework of FIG. 3, the operation flow itself within each processing step plays a role of processing execution adjustment means (702). The overwriting image determining means (703) is in the determining step (301), the duplication necessity determining means (704) is in the determining step (302), the determining step (303) and the copy creating step (304) is the image processing execution managing means ( 705) corresponds to the step in the flow after step (305).

In FIG. 4, after entering the processing step, in step (301), it is determined whether the image to be overwritten (output) by the processing step is a sharing target image. Determine if is necessary. When the shared image needs to be copied, the following two conditions are satisfied, and the determination is made in steps (302) and (303).
(1) The last process that outputs the shared image must have a pair (step (302))
(2) The other processing process in the pair may use the shared image (step (303))

  If it is determined that it is necessary, a copy of the image is created in step (304) and the process proceeds to the processing body. If it is determined that it is not a sharing target, or if it is a sharing target but it is determined that duplication is unnecessary, the process proceeds to the processing body as it is.

  After shifting to the processing main body, in step (305), it is determined whether or not the processing process is a processing process to be shared. If it is not the sharing target process, the process is executed as usual in step (306). If it is determined that the process is to be shared, it is checked in step (307) whether the paired process has already been executed. If not, the process is executed normally in step (308) and executed. If already completed, the processing result already executed in step (309) is reused as the result of its own processing.

  All the determinations described above are determined by referring to the image operation means (310) (C01 to C09).

  FIG. 5 shows the registered contents of the image operation means according to an embodiment of the present invention. The image operation means (310) includes an image information management means (401), a processing step information management means (402), a pair information management means (403), and an image duplication means (404).

  The image information management means (401) holds two pieces of information of shared information (401-1) and final processing step number (401-2) for each image number. The shared information (401-1) is information for determining whether or not the image is a share target image, and the final process step number (401-2) stores the process step number over which the target image was last overwritten. To do.

  The processing step information management means (402) holds information indicating whether or not the processing step number is a sharing target. Specifically, the pair number (402-1) may be held. If the pair number is registered, the processing step is a sharing target. If the pair number is not registered, it can be determined that the processing number is not the sharing target. The pair number is also used for reference to the pair information management means (403). Details of the pair information management means (403) will be described later. The image duplicating means (404) creates a copy of the designated image on the storage medium (311) according to the request of the processing flow in the processing step.

  The pair information management means (403) has three pieces of information, that is, the maximum number of shares (403-1), the number of shares counter (403-2), and the alternative image number (403-3) for each pair. The maximum number of shares (403-1) indicates the maximum number of shared predetermined images and is registered as a constant. The sharing number counter (403-2) is a variable indicating the number of times of actual sharing, and is registered with an initial value of 0. If the sharing count counter (403-1) is smaller than the maximum sharing count (403-1), the image duplicating means (404) duplicates the image. Not performed. The alternative image number (403-3) indicates the shared image number, and the initial value is empty. The substitute image number (403-3) also serves as information indicating whether the pair has already been executed. This is because there is a necessary and sufficient condition that the paired processing steps are not executed and that the alternative image number is not registered.

  FIG. 6 shows a simple example having two detection algorithms as an example of the detection algorithm, and the movement of the image operation means will be described with reference to this. An elliptical frame indicates an input / output image, and a rectangular frame indicates a processing step. For example, the detection algorithm A (501) receives the image 1 from the image input, and outputs the image 2 with the image 1 as an input in the processing step 1. Next, the processing result 2 is overwritten on the image 2 by processing the image 2 as an input. The processing step number (F01) is assigned to the processing step 1 of the detection algorithm A (501), and the processing step number (F02) is assigned to the processing step 2.

  The detection algorithm B includes processing step 1 and processing step 3. In processing step 1 of the detection algorithm B (502), the image 1 is input and the image 3 is output. In processing step 3, the result of processing with image 3 as an input is overwritten on image 3. The processing step number (F03) is assigned to the processing step 1 of the detection algorithm B (502), and the processing step number (F04) is assigned to the processing step 3. Images 1, 2 and 3 are assigned image numbers (I01), (I02) and (I03), respectively.

  Now, since the process step numbers (F01) and (F03) are exactly the same process, it is assumed that the process is recognized as a sharing target process process by analysis. A pair number (P01) is assigned to this pair. That is, the images to be shared are the image numbers (I02) and (I03) (however, the memory addresses are different). The symbols and numerical values shown in FIG. 5 are the initial registration contents for operating these two detection algorithms, and the maximum number of shares is 1.

  Here, it is assumed that the detection algorithm A (501) and the detection algorithm B (502) are allocated to different tasks by multitasking or the like and operate asynchronously. Asynchronous here means that the order in which each processing step operates is not uniquely determined between algorithms. For example, it may operate in the order of processing step numbers (F01)-(F02)-(F03)-(F04), or in the order of processing step numbers (F01)-(F03)-(F04)-(F02). May work. Hereinafter, an example of the operation in the former order will be described with reference to FIGS. 4 and 5. For simplicity, it is assumed that task switching does not occur inside the processing process.

-Operation of processing step number (F01) After inputting an image, first, the processing step number (F01) operates. After entering the processing step 1, the image operation means (310) is referred to in the determination step (301) to determine whether the overwritten image, that is, the output image (I02) is a sharing target. The registered content of the image number (I02) of the image information management means (401) is referred to (C01), and it can be seen from the shared information (401-1) that it is a sharing target. In the determination step (302), whether or not the processing has been completed is acquired from the image management means (C02). Since the final processing step number (401-2) is not registered, it is determined that the processing has not been performed. )

  In the common processing step (305), whether or not the processing step number (F01) is a shared processing step is acquired by referring to (C05) the image operation means (310). In the processing step information management means (402), the registered item of the processing step number (F01) is referred to, and since the pair number (402-1) has data, it is determined as a shared processing step, and the pair number (P01) is received. Based on the pair number (P01), in the determination step (307), the pair information management means (403) is referred to (C06), and it is confirmed that the alternative image number (403-3) has not been registered. Is determined to be unexecuted. Then, in step (308), the following information is reflected on the image operation means (310), and then the process ends.

(1) The final process step number (401-2) of the image number (I02) of the image information management means (401) is overwritten with the process step number (F01).
(2) Overwrite the alternative image number (403-3) of the pair number (P01) of the pair information management means (403) with the image number (I02).
FIG. 8 shows a state where the state shown in FIG. 5 is updated by the above-described overwriting.

-About operation | movement of a process process number (F02) A process process number (F02) operate | moves following said process process number (F01). After entering the processing step 2, it is determined in a determination step (301) whether the overwritten image, that is, the output image (I02) is a sharing target. Here, it is determined that the process is to be shared, and it is determined that the process is completed by the process step number (F01) from the registered content of the final process step number (401-2). Further, it is determined from the pair number (402-1) of the processing process information management means (402) that this is a shared processing process.

  In the determination step (303), the pair information management means (403) confirms (C03) whether other processing uses the number to be shared. Since the value 0 of the sharing number counter (403-2) of the pair information management means (403) is smaller than the value 1 of the maximum sharing number (403-1), it is determined that copying is necessary, and in step (304) A copy request (C04) is issued to the image operation means (310). Upon receiving the request, the image operating means (310) copies the contents of the image number (I02) to the image number (I04) by the DMAC, CPU or the like by the image duplicating means (404). Then, the substitute image number of the pair information management means (403) is changed to (I04).

In the common processing step (305), it is determined whether or not the processing step number (F02) is a sharing target processing step, and it is understood that it is not a sharing target by referring to the processing step information management means (402). The process is executed in step (306), and the final process step number (401-2) of the image number (I02) of the image information management means (401) is overwritten (C07) by the process step number (F02).
FIG. 9 shows the registered contents of the image operation means after the above operation.

-About operation | movement of process process number (F03) Process process number (F03) operate | moves after said process process number (F02). After entering the processing step 1, it is determined in a determination step (301) whether the overwritten image, that is, the output image (I03) is a sharing target. Although it is determined as a sharing target in the shared information (401-1) of the image information management means (401), it is determined as unprocessed by confirmation of the final process step number (401-2) in the determination step (302). Then, the process proceeds to the determination step (305). In the determination step (305), the processing process information management means (402) determines that the process is a sharing target processing process. In step (307), it is determined whether or not the pair has already been executed. Since the data (I04) has already been registered in the alternative image number (1001) of the pair information management means (403), it is confirmed that the pair has already been executed. In step (309), a reuse request (C09) is transmitted to the image operation means.

  Upon receiving the reuse request (C09), the image duplicating means (404) refers to the alternative image number (403-3) of the pair information managing means (403), and changes the image number (I04) to the image number (I03). make a copy. At this time, the value of the sharing number counter (403-2) is incremented by 1 to be 1. Through the above procedure, the processing step of the processing step number (F03) is completed without executing the processing in the processing step 1. However, after that, the final processing step number (401-2) of the image number (I03) of the image information management means (401) is overwritten on the pair number (402-1) of the processing step number (F03). FIG. 10 shows the registered contents of the image operation means after the above operation. The changed locations are the final process step number data (1101) and the shared number counter data (1102).

-About operation | movement of process process number (F04) And process process number (F04) operate | moves after said process process number (F03). After entering the processing step 3, it is determined in a determination step (301) whether the overwritten image, that is, the output image (I03) is a sharing target. The image information management means (401) determines that the image is to be shared, and the processing process number (F03) determines that the image has been processed. It is judged from the processing step information management means (402) that the processing step number (F03) is a shared processing step.

  In the determination step (303), the pair information management unit (403) confirms whether other processing uses the number to be shared, and the value 1 of the sharing number counter (403-2) of the pair information management unit (403) Is equal to the value 1 of the maximum sharing count (403-1), it is determined that replication is not necessary, and replication is not performed.

  In the next determination step (305), it is determined that the processing step number (F04) is not a shared processing step. Therefore, in step (306), the processing is executed as usual and the final processing of (I03) of the image information management means is performed. The process number is overwritten with the process process number (F04), and the process process 3 is exited. FIG. 11 shows the registered contents of the image operation means at this time. The changed part is the data (1201) of the final processing step number (401-2).

  Thereafter, even if further processing steps exist in the detection algorithm A and the detection algorithm B, duplication does not occur even if the image number (I02) and the image number (I03) are overwritten if there is no pair. The reason is that the final processing step number (401-2) of each image does not correspond to the shared processing step.

  In this embodiment, the image sharing side uses the original image, and the image sharing side uses the copied image. However, the image sharing side uses the alternative image number, and the sharing side uses the original image. You can use it. In that case, for example, one intermediate value may be inserted between the correspondence between the image number and the address in the memory space. This intermediate value is provided as an intermediate number (801) in the image information management means. (I'01) to (I'04) are intermediate numbers of images. For example, the image number is a number that the user can see on the program, and the intermediate number indicates the location of the image where the storage medium (311) is actually arranged. FIG. 7 shows this embodiment. The registered value in each management means is an initial value. When changing to this mechanism, the change from the description with reference to FIG. 5 is the operation of the process step number (F02). The operation of the processing step number (F02) is performed as described above, and the intermediate number corresponding to the image number (I02) is changed to the image number (I'04), and the intermediate number of the image number (I04) is changed to (I Change to '02). In this way, since the images actually handled on the storage medium are exchanged, the sharing side can use the original image.

Claims (7)

An image processing apparatus composed of a plurality of image processes composed of continuous processing steps, and when there is an identical processing step between the plurality of image processings, the output image data of the first processing step is used for the second and subsequent times. In an image processing apparatus that is reused for the same processing step,
For each image, image information management means for holding information on whether or not to reuse,
Overwrite image determination means for determining whether the image to be overwritten by the processing step is an image to be reused using the information;
Image duplicating means for creating a duplicate of the image data prior to overwriting only when the image is to be reused;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
In addition to storing the maximum number of times of sharing that is the number of times of reuse, a sharing number counter that counts the number of times of actual reuse is provided,
The image processing apparatus, wherein the image duplicating unit duplicates an image until the number of times of the sharing number counter reaches the maximum number of times of sharing. The image processing apparatus according to claim 2,
The image processing apparatus further includes a pair information management unit that has a plurality of pair numbers representing processing process groups in which images to be reused are equal, and stores the maximum number of sharing times and the number of times counted by the sharing number counter for each pair number An image processing apparatus. The image processing apparatus according to claim 3,
The image information management means includes means for storing a final processing step number that lastly overwrites the image for each image to be reused,
Only when the number of times of the sharing counter stored in the pair information management means is smaller than the maximum number of sharing, with reference to the case where the pair number is assigned for the processing step of the final processing step number An image processing apparatus for duplicating the image. The image processing apparatus according to claim 4,
An image processing apparatus comprising: a substitute image holding unit that associates images before and after the duplication with respect to each pair number. An image processing apparatus composed of a plurality of image processes composed of continuous processing steps, and when there is an identical processing step between the plurality of image processings, the output image data of the first processing step is used for the second and subsequent times. In an image processing apparatus that is reused for the same processing step,
Image information management means for storing shared information and final processing step number for the image,
For process step numbers, process step information management means for storing pair numbers,
For the pair number, an image operation means comprising a pair information management means for storing a maximum number of times of sharing, a number of times of sharing counter and an alternative image number, and an image duplicating means; An image processing apparatus comprising processing execution adjustment means. An image processing method comprising a plurality of image processes comprising continuous processing steps, wherein when there is an identical processing step between the plurality of image processings, the output image data of the first processing step is used for the second and subsequent times. In an image processing method that is reused in the same processing step,
For each image, storing information about whether to reuse,
Determining whether the image to be overwritten by the processing step is an image to be reused using the information; and
Creating a copy of the image data prior to overwriting only if the image is to be reused; and
An image processing method comprising:

Images