JP2006178857A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor capable of extracting a particular object in a moving image on the basis of a processing program obtained by combining respective image filters in the shape of a tree structure, especially, a particular object accompanied with temporal change and displacement and to further provide an image processor with high versatility capable of easily obtaining such a processing program. <P>SOLUTION: The image processor 1 for performing image processing of an image picked up by an image pickup device 21 to extract a particular object in the image is provided with an image processing part 3 for performing image processing of a plurality of types of input images t-1 to t-k picked up by the image pickup device 21 to form an output image with the particular object extracted on the basis of the processing program obtained by combining the image filters in the shape of a tree structure. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus capable of extracting a specific target from an image.

  In recent years, an image input means such as a TV camera or a CCD camera is used to capture an image of a subject, a landscape, etc., and the obtained moving image is subjected to image processing. Research on image processing apparatuses and image processing methods for extracting the movements and the like has been actively conducted (for example, see Patent Documents 1 to 5).

  Such an image processing apparatus, for example, in the field of automobiles or the like, captures a front landscape with a CCD camera or the like attached to the vehicle, extracts pedestrians and other vehicles from the moving image, and collides with them. In the field of artificial intelligence robots, etc., robots discover and measure other moving objects while observing the environment with a built-in camera. Research and practical application are being made in various fields, such as determining the behavior of a robot in response thereto (see Patent Document 4).

  In order to extract a specific target from a moving image, in such an image processing apparatus, for example, an input image obtained by two CCD cameras or the like separated in the horizontal direction is subjected to image processing and the contour of the specific target Image processing methods such as extracting a part, calculating an optical flow from an input image and extracting a specific target, or collating with a model registered in a database by pattern matching processing to estimate the specific target. It has been.

  However, these methods usually require a lot of work to construct a processing program, and a processing program must be constructed for each specific target object. Therefore, there has been a demand for an image processing method and an image processing apparatus that can easily construct a processing program and can obtain a highly versatile processing program.

  On the other hand, in the field of image processing for still images, an image processing technique (ACTIT) for performing image processing on an input image based on a processing program in which various image filters are combined in a tree structure as shown in FIG. 19 has been recently proposed. (See Non-Patent Document 1).

  Specifically, for example, an input image of a document composed of print characters and handwritten characters is subjected to image processing by a processing program as shown in FIG. 19, and an output image in which only print characters are extracted is output. This is an image processing technique for performing image processing such as obtaining an output image by extracting only a mesh pattern at a boundary portion between cells from a microscope image in a state where a plurality of corneal endothelial cells are connected.

Non-Patent Document 1 further proposes to adopt a technique of genetic programming (abbreviated as GP) in order to automatically optimize the combination of the various image filters.
JP-A-5-265547 Japanese Patent Laid-Open No. 10-11585 JP 2002-83297 A JP 2001-84383 A JP-A-9-271014 Shinya Aoki, 1 other person, “Automatic construction method of tree-structured image conversion ACTIT”, Journal of the Institute of Image Information and Television Engineers, The Institute of Image Information and Television Engineers, 1999, Vol. 888-894

  Therefore, it is expected that the image processing technique described in Non-Patent Document 1 is applied to the above-described problem of extracting a specific target such as a moving object from the moving image.

  However, as described above, this image processing technique performs image processing on a still image. Specifically, the same still image is used as the “input image” of the processing program shown in FIG. It is assumed that is repeatedly input.

  Therefore, an object of the present invention is to extend this image processing technique so that the same image processing technique can be applied to a moving image, and based on a processing program combining various image filters in a tree structure, An object of the present invention is to provide an image processing apparatus capable of extracting a specific target, and particularly capable of extracting a specific target with a temporal change or displacement. It is another object of the present invention to provide a highly versatile image processing apparatus that can easily obtain such a processing program.

In order to solve the above problem, an image processing apparatus according to claim 1,
In an image processing device that performs image processing on an image captured by an imaging device and extracts a specific target from the image,
An image processing unit for performing image processing on a plurality of types of images captured by the imaging device based on a processing program in which image filters are combined in a tree structure, and forming an output image from which the specific target is extracted It is characterized by providing.

  According to the first aspect of the present invention, the configuration of the tree-structured processing program included in the image processing unit of the image processing apparatus is not a configuration having only the same still image as a terminal symbol as in the prior art. A tree-structured processing program using a plurality of types of images as terminal symbols.

  According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the plurality of types of images are a plurality of images captured at time intervals by the imaging device. .

  According to the second aspect of the present invention, the tree structure processing program is configured such that a plurality of moving image frames captured at time intervals are used as a plurality of types of input images. .

  According to a third aspect of the present invention, in the image processing device according to the first aspect, the plurality of types of images are inter-frame difference images or edges of a plurality of images captured at time intervals by the imaging device. It is an extracted image.

  According to the invention described in claim 3, the tree structure processing program uses, as the plurality of types of input images, inter-frame difference images or edge extraction images of a plurality of images captured at time intervals. Configured to be.

  According to a fourth aspect of the present invention, in the image processing apparatus according to any one of the first to third aspects, the image processing apparatus includes a processing program forming unit for forming the processing program. A processing program is formed by genetic programming using the plurality of types of images, target images, and weight images.

  According to the fourth aspect of the present invention, the image processing apparatus should output the moving image captured by the imaging device and the formed processing program when the moving image is input in the processing program forming unit. Genetic programming using a target image given as an image and a weighted image for weighting the pixel-by-pixel difference between the luminance value of the output image and the luminance value of the target image (exactly the absolute value of the difference) The processing program is automatically formed by the above.

  According to a fifth aspect of the present invention, in the image processing apparatus according to the fourth aspect, the weighted image has a ratio of the weight of the extraction region to the weight of the non-extraction region. It is set so that it may become the ratio of the reciprocal of area ratio.

  According to the invention described in claim 5, the weighting by the weighted image is different between the extracted region and the non-extracted region of the image, and in particular, the weight ratio is the reciprocal of the area ratio of the extracted region and the non-extracted region. So that the ratio of

  According to a sixth aspect of the present invention, in the image processing device according to the fourth or fifth aspect, the processing program forming unit uses a plurality of learning sets including the plurality of types of images, target images, and weight images. And forming a processing program.

  According to the invention described in claim 6, in order to form one processing program, not only one learning set composed of a plurality of types of images, target images and weight images but also different learnings with different scenes, for example. Use multiple sets.

  According to a seventh aspect of the present invention, in the image processing device according to any one of the fourth to sixth aspects, the fitness used for the genetic programming in the processing program forming unit is in the processing program. It is calculated so that it may take a small value, so that the number of nodes is large.

  According to the seventh aspect of the present invention, in genetic programming, a processing program having a large number of nodes is easily deceived.

  The invention according to claim 8 is the image processing apparatus according to claim 7, wherein the ratio of the number of nodes to the fitness changes according to the number of generations of the evolution process in genetic programming. It is characterized by that.

  According to the invention described in claim 8, in the invention described in claim 7, the rate at which the fitness is decreased as the number of nodes increases is changed according to the evolution process in genetic programming.

  According to a ninth aspect of the present invention, in the image processing apparatus according to any one of the fourth to eighth aspects, the fitness used for the genetic programming in the processing program forming unit is in the processing program. The two-input image filter is calculated so as to take a larger value as the number of nodes is larger.

  According to the ninth aspect of the present invention, in genetic programming, the fitness of a processing program having a large number of nodes of a two-input image filter is increased to make it difficult to be deceived.

  According to a tenth aspect of the present invention, in the image processing apparatus according to the ninth aspect, the ratio of the number of nodes of the two-input image filter to the fitness is changed according to the number of generations of the evolution process in genetic programming. It is comprised by these.

  According to the invention described in claim 10, in the invention described in claim 9, the rate at which the fitness is increased as the number of nodes of the two-input image filter increases is changed according to the evolution process in genetic programming.

  According to an eleventh aspect of the present invention, in the image processing apparatus according to any one of the first to tenth aspects, the processing program is formed by combining a plurality of processing programs. To do.

  According to the eleventh aspect of the present invention, a large-scale processing program is formed by combining a plurality of processing programs formed artificially or processing programs formed by genetic programming.

  According to a twelfth aspect of the present invention, in the image processing apparatus according to the eleventh aspect of the present invention, an output image is formed by nonlinear superposition of processing by the plurality of processing programs. .

  According to the invention described in claim 12, in the image output by the large-scale processing program according to the invention described in claim 11, a large number of each processing program constituting the large-scale processing program has an output luminance value. The pixel portion has a large output luminance value, and the pixel portion having only a small number of output luminance values has a small or zero output luminance value.

  According to a thirteenth aspect of the present invention, in the image processing device according to any one of the first to twelfth aspects, the image filter includes a mask filter.

  According to the invention of claim 13, a mask filter is used as a non-terminal symbol constituting a tree-structured processing program, or a combination of processing programs in a large-scale processing program configured by combining a plurality of processing programs is masked. A filter can be used.

  According to a fourteenth aspect of the present invention, in the image processing device according to any one of the first to thirteenth aspects, an output image is provided that includes a display unit for displaying an image and is formed based on the processing program. Is superimposed on the input image displayed on the display unit and displayed.

  According to the fourteenth aspect of the present invention, an output image colored in red is superimposed and displayed on an input image displayed as a black and white image, for example, on the display unit of the image processing apparatus.

  According to the first aspect of the present invention, the configuration of the tree-structured processing program included in the image processing unit of the image processing apparatus is not a configuration having only the same still image as a terminal symbol as in the prior art. A tree-structured processing program using a plurality of types of images as terminal symbols. For this reason, the conventional ACTIT (see Non-Patent Document 1) image processing technique is expanded so that the ACTIT technique can be applied to moving images having different images for each frame.

  At the same time, by comparing each input image and performing difference processing or logical product processing, it becomes possible to perform image processing that takes into account factors such as the displacement of the position of a specific target between the images, It is possible to extract a specific object with temporal change or spatial displacement in the image.

  According to the second aspect of the present invention, the tree structure processing program is configured such that a plurality of moving image frames captured at time intervals are used as a plurality of types of input images. Therefore, by performing the difference processing of each frame of the moving image, it is possible to more accurately extract a specific target that is particularly accompanied by a temporal change or displacement.

  According to the invention described in claim 3, the tree structure processing program uses, as the plurality of types of input images, inter-frame difference images or edge extraction images of a plurality of images captured at time intervals. Configured to be. Therefore, the processing program can directly extract a specific target with a temporal change or the like based on a time differential data image (that is, an inter-frame difference image) of each frame of the moving image. .

  According to the fourth aspect of the present invention, the image processing apparatus should output the moving image captured by the imaging device and the formed processing program when the moving image is input in the processing program forming unit. Genetic programming using a target image given as an image and a weighted image for weighting the pixel-by-pixel difference between the luminance value of the output image and the luminance value of the target image (exactly the absolute value of the difference) The processing program is automatically formed by the above. Therefore, in addition to the effects of the inventions described in the above claims, a processing program can be obtained very easily by providing an appropriate target image and weight image.

  Moreover, the specific object which should be extracted can be changed easily by changing a target image and a weight image. In other words, the genetic programming described above was used only by changing the target image and the weight image without manually constructing a processing program for the extraction every time the specific target is changed as in the past. It is possible to easily construct a processing program in the same procedure using the method as it is.

  According to a fifth aspect of the present invention, in the fourth aspect of the present invention, the weighting by the weighted image is different between the extraction region and the non-extraction region of the image, and in particular, the weight ratio is the extraction region. And it sets so that it may become the ratio of the reciprocal of the area ratio of a non-extraction area | region. Although it is possible to use the same weight for all the pixels of the weighted image, in this case, it is easy to cause a result in which the degree of coincidence of the extracted area and the non-extracted area having a larger area is more important. By setting the weight ratio to be the reciprocal of the area ratio of the extraction region and the non-extraction region, it is possible to obtain a processing program in which the degree of coincidence between the extraction region and the non-extraction region is equally important Become.

  According to the invention described in claim 6, in order to form one processing program, not only one learning set composed of a plurality of types of images, target images and weight images but also different learnings with different scenes, for example. Use multiple sets. Therefore, in addition to the effects of the inventions described in the above claims, it is possible to avoid the phenomenon of extraction of a target at a specific position in an image that easily falls when only one set is used as a learning set. Specific objects can be extracted more accurately from images that were not used in the learning set for dynamic programming, and it is possible to obtain a versatile processing program that accurately extracts only people from moving images It becomes.

  According to the invention described in claim 7, in the genetic programming, the processing program having a large number of nodes is easily deceived, so that the processing program is in an over-learning state in addition to the effects of the inventions described in the above claims. It becomes possible to avoid becoming. That is, the processing program may extract more limited objects (for example, pedestrians wearing dark clothes for pedestrians in general) as the number of nodes increases. It is possible to effectively prevent the state processing program from being formed.

  According to the invention described in claim 8, in the invention described in claim 7, the ratio of decreasing the fitness as the number of nodes increases is changed according to the evolution process in genetic programming, so that the number of nodes increases. It is possible to freely decide at which stage of the evolution process the processing program is easily deceived. As genetic programming evolves, even if the number of generations increases, the maximum fitness value may not increase and stagnation may occur. When such stagnation occurs, the ratio may be changed. This increases the possibility of obtaining a more optimized processing program.

  According to the invention described in claim 9, in genetic programming, the fitness of a processing program having a large number of nodes of a two-input image filter is increased to make it difficult to be deceived. In addition to the effect, it is possible to increase the diversity of images input to the processing program, and to obtain a processing program that can extract a specific target more accurately.

  According to the invention described in claim 10, in the invention described in claim 9, the rate of increasing the fitness as the number of nodes of the two-input image filter increases is changed according to the evolution process in genetic programming. It is possible to freely determine at which stage of the evolution process the processing program having a large number of nodes of the 2-input image filter is likely to survive. As genetic programming evolves, even if the number of generations increases, the maximum fitness value may not increase and stagnation may occur. When such stagnation occurs, the ratio may be changed. This increases the possibility of obtaining a more optimized processing program.

  According to the invention described in claim 11, the invention according to each of the above claims is formed by forming a large-scale processing program by combining a plurality of processing programs formed artificially or processing programs formed by genetic programming. In addition to the effect, it is possible to more accurately extract the target specific object. Furthermore, when a processing program is obtained by genetic programming, the search space for the solution program increases exponentially as the number of image filters (non-terminal symbols) constituting the processing program increases, and a huge search is required. Although the possibility of falling into a local solution increases, it becomes possible to obtain a highly versatile processing program that can extract a specific target more easily and accurately by combining a plurality of processing programs in this way.

  According to the invention described in claim 12, in the image output by the large-scale processing program according to the invention described in claim 11, a large number of each processing program constituting the large-scale processing program has an output luminance value. The pixel portion has a large output luminance value, and the pixel portion having only a small number of output luminance values has a small or zero output luminance value. Therefore, the output image by the large-scale processing program becomes clearer, and noise that should not be extracted can be effectively removed.

  According to the invention of claim 13, a mask filter is used as a non-terminal symbol constituting a tree-structured processing program, or a combination of processing programs in a large-scale processing program configured by combining a plurality of processing programs is masked. A filter can be used. Therefore, in addition to the effects of the inventions described in the above claims, unnecessary noise is removed from the output image of a processing program or a large-scale processing program, or the output image is divided into different areas. It is possible to assign an output image of the processing program and form an output image.

  According to the fourteenth aspect of the present invention, the image processing apparatus includes a display unit, and the display unit displays, for example, an input image displayed as a black-and-white image superimposed on an output image colored in red. Therefore, in addition to the effects of the inventions described in the above claims, a specific object in the input image can be clearly displayed.

  Hereinafter, embodiments of the image processing apparatus of the present invention will be described with reference to the drawings.

  In the present embodiment, an image processing apparatus will be described in which an image processing apparatus is mounted on an automobile vehicle and a pedestrian is extracted from a landscape image in front of the vehicle.

  FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to the present embodiment. The image processing apparatus 1 includes an image input unit 2, an image processing unit 3, a display unit 4, a memory 5, and processing program formation. A unit 6 and an input unit 7 are provided. In the present embodiment, as the image processing apparatus 1, a computer configured by connecting a CPU, a RAM, a ROM, an input / output interface, and the like through a BUS can be used.

  The image input unit 2 includes an imaging device 21 that can convert a captured image into an electrical signal. As the imaging device 21, for example, a CCD camera using a solid-state imaging device such as a charge coupled device (CCD) is used. It is done. In the present embodiment, the imaging device 21 of the image input unit 2 is attached to the inside of the windshield near the rear mirror of an automobile vehicle (not shown) so that the front can be imaged, and is 1/30 like a normal TV image. The vehicle front is imaged every second and the input image is transmitted to the image processing unit 3.

  In the present embodiment, the unit of the input image transmitted at regular time intervals is referred to as one frame. That is, in this embodiment, the image processing unit 3 is configured to transmit an input image of 30 frames per second from the image input unit 2.

  The image processing unit 3 is connected to a display unit 4 having a monitor and a memory 5, and the image processing unit 3 transmits the input image transmitted from the image input unit 2 to the display unit 4 to be used as a monitor. Simultaneously with the display, the input images are temporarily stored in the memory 5 sequentially.

  The image processing unit 3 stores a processing program in which various image filters are combined in a tree structure, and the image processing unit 3 performs image processing according to the processing program to form an output image. It is configured.

  Here, the structure of the processing program will be described. As shown in FIG. 2, the processing program is a program in which various image filters (see FIG. 3) are combined in a tree structure, and a plurality of input images t, t−1,. k is an integer greater than or equal to 1), that is, k + 1 different input images t, t−1,..., t−k are subjected to image processing by each image filter to form an output image. Yes.

  In this embodiment, as the input images t, t−1,..., Tk, the input image t at the current time t and the input images t, t−1,. The program is input to the processing program, and k = 3 and M = 1 are set. That is, as shown in FIG. 4A, the input image t at the current time t and every previous frame, that is, the image was taken at a time interval of 1/30 second from the current time t. Input images t, t-1, t-2, and t-3 for four consecutive frames are read from the memory 5 and input to the processing program.

  The values of k and M can be set as appropriate. For example, if k = 2 and M = 3 are set, as shown in FIG. A total of three images, i.e., input images t, t-1, and t-2 for each frame, are read out and input to the processing program. It is also possible to select a plurality of different types of input images by other selection methods and input them to the processing program.

  The processing program of the present embodiment uses a general image filter as shown in FIG. 3 in order to improve the calculation speed. However, an image filter specialized in function according to the purpose is added. It is also possible to do.

  As described above, the image processing apparatus 1 according to the present embodiment extracts a pedestrian as a specific target from a landscape image in front of the vehicle, and the processing program also extracts a pedestrian from the input image T. Is configured to do. That is, the input images t, t-1, t-2, and t-3 (see FIG. 5) for the four consecutive frames described above (see FIG. 5; however, in FIG. 5, the input image t (A) and the input image t-3 (B ) Only is input to the processing program, the processing program performs image processing with each image filter to form an output image in which a specific target including a pedestrian as shown in FIG. 6 is extracted. ing. In the output image of FIG. 6, the luminance value of the hatched pixel is 0.

  Furthermore, in the present embodiment, the output image formed in this way is configured to be displayed superimposed on the input image t displayed on the monitor of the display unit 4. That is, as described above, the input image t transmitted from the image processing unit 3 is displayed on the monitor of the display unit 4, and further, as shown in FIG. 7, the output formed by the processing program on the input image t. The images are displayed so as to overlap each other. At that time, the input image t is displayed as a black and white image, and the pixel portion (hatched portion in the figure) having a positive luminance value in the output image is displayed in red.

  At that time, it is also possible to perform the image processing by the mask filter as shown in FIG. 8 on the output image of the processing program. For example, the red colored portion F of the vehicle front portion of FIG. Since the colored portion T of the tree above the image is an unnecessary colored portion, it is possible to perform image processing using a mask filter so as not to display the color of those portions.

  By the way, the processing program can be artificially constructed and given to the image processing unit 3. In the processing program having a tree structure as shown in FIG. 2, for example, one-input or two-input image filters as shown in FIG. 3 are arbitrarily combined in a maximum of 40 ranges, and input images t and t−1 are combined. , T-2, and t-3 can be arbitrarily searched for all tree-structured processing programs.

  In this embodiment, the processing program forming unit 6 connected to the image processing unit 3 is automatically formed using a genetic programming technique. FIG. 9 is a block diagram showing the configuration of the processing program forming unit. The processing program forming unit 6 includes an initial individual generating unit 61, fitness evaluation units 62 and 66, a parent selecting unit 63, and a crossing unit 64. , A mutation means 65 and an end determination means 67 are provided.

  The initial individual generation means 61 sets a processing program having a tree structure as shown in FIG. 2 in accordance with a processing program formation instruction from the input unit 7 (see FIG. 1) such as a keyboard and a mouse. And a fixed number (100 individuals in this embodiment) are randomly generated within the range of the values of M and M.

  As a rule for randomly generating a processing program, in this embodiment, an image filter (that is, a non-terminal) among nodes constituting a processing program having a tree structure in an evolution process until an optimized processing program is obtained as well as an initial individual. The number of symbols) is set so as not to exceed 40 at the maximum. The image filter is randomly selected from the image filters shown in FIG. In addition to the various image filters shown in FIG. 3, the selected image filter can include a mask filter as shown in FIG.

  Further, as described above, in this embodiment, k = 3 and M = 1 are set, and four consecutive frames imaged at a time interval of 1/30 seconds retroactive from a certain time t. The input images to be input to the processing program are arbitrarily selected from the input images t, t-1, t-2, and t-3. That is, it is not necessary to use all four types of input images t, t−1, t−2, and t−3 as input images of the processing program. Some of the initial individuals include, for example, the input image t and the input image t−. It is allowed to include a processing program that uses only two types of 2 or only the input image t-3.

  An fitness evaluation means 62 is connected to the initial individual generation means 61 so that the initial individual of each processing program generated by the initial individual generation means 61 is transmitted to the fitness evaluation means 62.

  The fitness evaluation means 62 executes a simulation for inputting an input image for each processing program and obtaining an output image, and compares the output image obtained by the simulation with the target image to determine the fitness E of each processing program. It is configured to be calculated based on the following equation (1).

  Here, the target image is an image to be output by the optimized processing program. In the present embodiment, since the purpose of the processing program is to extract pedestrians from a landscape image in front of the vehicle, only the pedestrians are outlined from the input image t (for example, see FIG. 5A). An image (see FIG. 10) in which the extracted area (luminance value 255) is set and the other part is the non-extracted area (luminance value 0) is transmitted to the fitness evaluation means 62 as a target image.

  The weighted image is an image in which the weight for weighting the distance | OT−T | between the output image (O) and the target image (T) for each pixel is defined for each pixel. Is appropriately determined according to the purpose of the processing program to be constructed. Normally, a large value is set in a pixel area where the output image is strongly required to match the target image, and a small value is set in a pixel area where the match between the output image and the target image is not strongly required.

  In this embodiment, since the purpose is to extract pedestrians and not to extract others, the output image matches the target image both in the target image extraction region and in the non-extraction region. Strongly required. However, if the weight is set to the same value in all the images, the area ratio of the pixel area occupied by the pedestrian (that is, the extraction area) in the output image is smaller than the area ratio of the other pixel areas (that is, the non-extraction area) (area ratio). 12: 256), the contribution of the matching degree in the non-extraction region in the fitness evaluation may be too large.

  Therefore, in this embodiment, the weight image is the same as the target image (see FIG. 10) as shown in FIG. 11, and the ratio of the weight of the extraction region to the weight of the non-extraction region is the reciprocal of the respective area ratios. The weight (W) is set to be 1/12 and 1/256 in the extracted region and the non-extracted region, respectively, so that the ratio becomes. The weighted image is transmitted to the fitness evaluation means 62 together with the target image and the input images t, t−1, t−2, and t−3 read from the memory 5.

  In the fitness evaluation means 62, the fitness E of each processing program is calculated using a plurality of types of input images t, t-1, t-2, t-3, target images and weight images as described above. In the present embodiment, the processing program is simulated by using not only one set but also a plurality of sets (hereinafter referred to as a learning set) formed by combining an input image, a target image, and a weight image. ing.

  That is, for example, as shown in FIG. 12, together with a first learning set consisting of a plurality of types of input images t, t−1, t−2, and t−3 and corresponding target images and weight images at time t, A similar second learning set such as an input image (see FIG. 13A) at time ta before t, and a similar third learning such as an input image at time tb after time t (see FIG. 13B). A total of three sets (in this case, N = 3 in the equation (1)) is sent to the fitness evaluation means 62, and one set for each processing program is simulated three times in total. ΣW · | OT | / ΣW · 256 in the equation (1) is calculated to obtain the fitness E based on the equation (1).

  A parent selection means 63 is connected to the fitness evaluation means 62, and each processing program whose fitness E is calculated by the fitness evaluation means 62 is transmitted to the parent selection means 63.

  Based on the fitness E, the parent selection means 63 selects 100 individual processing programs to be left in the next generation by methods such as roulette selection, expected value selection, ranking selection, tournament selection, etc., and their processing. It is configured to multiply the program. Further, in the present embodiment, 100 individuals are selected by tournament selection, and the elite storage of the processing program having the maximum fitness E is simultaneously performed.

  The processing program for 100 individuals selected and expanded by the parent selection means 63 is transmitted to the next crossing means 64.

  In the crossover means 64, as shown in FIG. 14, the processing programs sent from the parent selection means 63 are paired by 2 individuals (referred to as parent programs 1 and 2), and randomly selected for each individual pair. The crossed portions (portions surrounded by dotted lines of the parent programs 1 and 2 in FIG. 14) are crossed with each other at a predetermined ratio to generate child programs 1 and 2. At that time, if the number of non-terminal symbols exceeding 40 is generated in two child programs, the crossover is canceled and the crossover part is selected at random in the original parent programs 1 and 2 again. Crossover is performed.

  In the present embodiment, the crossover means 64 is configured to perform one-point crossover as shown in FIG. 14, but in addition to this, for example, a multipoint crossover or uniform crossover is used. It is also possible to configure it to cross.

  A processing program that is a child program of 100 individuals generated by the crossover means 64 is sent to the next mutation means 65.

  The mutation means 65 is configured to generate node mutations, insertions, deletions, etc. at a predetermined rate for each processing program. At this time, if the number of non-terminal symbols in the processing program exceeds 40 due to the insertion of the node, the insertion is not performed, and the terminal symbols (that is, the input image t, etc.) and the non-terminal symbols (that is, the image filter) Mutations with are prohibited. In addition to this, for example, it is possible to perform a mutation such as translocation or duplication, and appropriate restrictions are added accordingly.

  A fitness evaluation means 66 is connected to the mutation means 65, and the processing program of 100 individuals generated by the mutation means 65 is sent to the fitness evaluation means 66. In the fitness evaluation means 66, processing similar to that of the fitness evaluation means 62 described above is performed, and the same first to third learning sets as those used in the fitness evaluation means 62 are used, and simulation is performed for each processing program. The fitness E is calculated based on the equation (1).

  The fitness evaluation means 66 is connected to an end determination means 67, and each processing program whose fitness E is calculated by the fitness evaluation means 66 and the fitness of the previous generation elite-stored by the parent selection means 63. Is sent to the end determination means 67, and it is determined whether or not the processing program formation in the processing program forming unit 6 is to be ended.

  In the present embodiment, the end determination unit 67 determines whether the number of generations of the evolution process has reached a preset end generation number Ge, and when it is determined that the end generation number Ge has been reached, at that time A processing program having the maximum fitness E is output to the image processing unit 3 as a solution, and the program formation is terminated. If the end determination unit 67 determines that the number of generations has not reached the end generation number Ge, the end determination unit 67 sends each processing program to the parent selection unit 63 and repeats the above-described processing procedure.

  In addition to this, for example, the end determination means 67 determines whether or not there is a processing program whose fitness has reached a preset target fitness Eq in each processing program, and has reached the target fitness Eq. If there is a processing program, the processing program may be output to the image processing unit 3 as a solution. In addition, the end determination means 67 is configured to store the maximum fitness value of each processing program, and when the maximum fitness value does not change during a predetermined number of generations, that is, the maximum fitness value. It is also possible to configure such that when the stagnation occurs, the procedure is terminated at that generation, and the processing program having the maximum fitness is output to the image processing unit 3 as a solution.

  In the processing program forming unit 6, a processing program optimized based on the above evolution process is formed, but a phenomenon called so-called over-learning may be seen in the obtained processing program. That is, in the case of this embodiment, instead of extracting pedestrians in general, more limited, for example, pedestrians wearing white clothes are not extracted, only pedestrians wearing dark clothes May be obtained.

  In order to avoid the occurrence of such overlearning, in this embodiment, in the fitness evaluation by the fitness evaluation means 62 and 66, the following formula (2) is further calculated from the fitness E calculated by the formula (1). Based on the above, the fitness E ′ in consideration of overlearning restriction is calculated. Therefore, in the present embodiment, the parent selection unit 63 and the end determination unit 67 compare and refer to the fitness E ′ in consideration of this overlearning restriction.

  Here, the coefficients a and b both take positive values. According to the equation (2), the fitness E ′ in consideration of overlearning restriction is calculated so as to take a smaller value as the number of nodes n (Node) in the processing program is larger, and the number of nodes of the two-input image filter. The larger the value of m (2input-node) is, the larger the value is calculated.

  The reason why the fitness E ′ considering the overlearning restriction is configured as in the above equation (2) is that the larger the number of nodes of the tree structure of the processing program, the more the target to be extracted becomes more overlearned. This is because, as the number of nodes is smaller, more general objects (in general, pedestrians in this embodiment) can be extracted, and versatility is improved.

  Further, when the fitness E ′ is reduced as the number of nodes is increased, the ratio of the two-input image filter in the tree structure of the processing program is reduced. For example, as in the present embodiment, However, there is a processing program that actually inputs fewer types of input images even if four types of input images (that is, input images t, t-1, t-2, and t-3) are allowed to be input. Since the tendency to be obtained becomes stronger, the fitness E ′ is configured to take a larger value as the number of nodes of the two-input image filter is larger.

  The coefficients a and b represent the ratio of the number of nodes to the fitness level E ′ and the ratio of the number of nodes of the two-input image filter to the fitness level E ′. The coefficients a and b are inherited in the processing program forming unit 6. It can be configured to change according to the number of generations in the evolutionary process of dynamic programming.

  For example, if the coefficients a and b both take a large value and take a small value with the generation when the number of generations is small, a processing program with a large number of nodes can be easily deceived (effect of a). There is a high possibility that a processing program containing a large amount of will survive (effect of b). Conversely, if the coefficients a and b are increased with the generation, the processing specialized for the learning set obtained in the early stage of evolution can be made simpler in the latter half of the evolution.

  Furthermore, when evolution progresses and the stagnation of the maximum value of fitness occurs, it is highly likely that a more optimized processing program can be obtained by artificially changing the values of the coefficients a and b.

  The processing program formed by the processing program forming unit 6 as described above is sent to the image processing unit 3 as described above. Further, in the present embodiment, as shown in FIG. 15, a plurality of processing programs 1 to n formed are combined to form and use one larger processing program.

  As a combination method, for example, a logical sum is obtained for each corresponding pixel of n output images obtained by the processing programs 1 to n, and a binarized image is output as an output image of a large-scale processing program. It is also possible to configure as follows. Further, a large-scale processing program as shown in FIG. 16 is configured by using the mask filter shown in FIGS. 8A and 8B, and the processing program 1 and the upper half of the output image are formed. It is also possible to display a result of image processing by the processing program 2 so as to form one output image.

  In the present embodiment, a large-scale processing program is constructed by combining six processing programs obtained by genetic programming in the processing program forming unit 6. In this large-scale processing program, in order to remove noise from the output image and display a stronger red color in pixels in which the image is extracted by many processing programs among the pixels of the output image, Assuming that the output result of the i-th processing program is di, the output luminance value D in each pixel of the output image is determined based on the non-linear superposition represented by the following equation (3).

  Here, in the present embodiment, n = 6 and p is set to 2. The threshold value K is a constant and is set to 127 in this embodiment. The values of p and K can be arbitrarily set. If the value of p is increased, the pixel from which the image is extracted can be displayed with more emphasis.

  Next, the operation of the image processing apparatus of this embodiment will be described.

  An imaging device 2 (see FIG. 1) of the image processing device 1 attached to the inside of the windshield of the automobile vehicle images a front landscape of the vehicle, and uses the image (see FIG. 5A) as an input image t as an image processing unit. 3 to send. The imaging device 2 repeats this operation every 1/30 seconds.

  When receiving the input image t from the imaging device 2, the image processing unit 3 transmits the input image t to the display unit 4 to be displayed on the monitor, and temporarily stores the input image t in the memory 5 sequentially. At the same time, the input images t-1, t-2, and t-3 stored in the memory 5 are read, and the input images t, t-1, t-2, and t-3 are converted into a tree structure. An output image is formed by inputting to the combined processing program, and the output image colored in red is superimposed on the input image t displayed as a monochrome image on the monitor of the display unit 4 and displayed.

  Here, as described above, the processing program may be artificially constructed, but may be formed in advance by genetic programming in the processing program forming unit 6.

  The procedure for forming the processing program in the processing program forming unit 6 is as described above, and FIG. 17 shows an example of a processing program as a solution formed by genetic programming in the processing program forming unit 6. In this case, the number of non-terminal symbols, that is, the number of image filters is 40, the number of input images among the terminal symbols is 15, and the number of output images is 1.

  Also, a large-scale processing program can be formed by combining the processing program shown in FIG. 17 and a processing program obtained in the same manner based on the equation (3). For example, when the input images t, t-1, t-2, and t-3 (see FIG. 5) are input to this large-scale processing program, specific objects including the pedestrian shown in FIG. 6 are extracted. When an output image is obtained and the output image colored in red is superimposed on the input image t displayed as a monochrome image on the monitor of the display unit 4, an image as shown in FIG. 7 is obtained.

  When the processing program formed in the processing program forming unit 6 is viewed, it can be observed that the processing is performed so that the processing by the differential filter is performed at an early stage of the processing of the image filter for the input image. Many. The purpose of the processing program in this embodiment is to capture a front landscape from a moving vehicle and extract a moving or stopped pedestrian from the image. This is considered to be because pedestrians are extracted from a plurality of time-series input images in which the gradual change slightly occurs.

  Therefore, as input images t, t−1, t−2, and t−3, instead of performing all image input as shown in FIG. 5, a plurality of images captured at time intervals by the imaging device It is also possible to configure to input an inter-frame difference image (that is, time differential data) of the image. Moreover, it is also possible to configure so that edge extraction images (that is, spatial differential data in each image) in each of the input images t, t-1, t-2, and t-3 are input.

  As described above, according to the image processing apparatus of the present embodiment, a plurality of types of input images t and t of a front landscape of a vehicle captured at time intervals in a processing program in which image filters are combined in a tree structure. By making it possible to input -1,..., Tk, it becomes possible to input a plurality of frames of a moving image into the processing program, and the moving image can be processed.

  Further, each frame of a moving image (for example, a landscape in front of a vehicle) is compared by various image filters such as a difference filter constituting a tree structure processing program and subjected to image processing such as a difference, so that the moving image It is possible to effectively form an output image in which a specific object (a pedestrian in the case of the present embodiment) accompanied by temporal changes and displacements is extracted.

  Furthermore, the processing program can be easily obtained by automatically forming the processing program by genetic programming in the processing program forming unit. Moreover, the specific object which should be extracted can be changed easily by changing a target image and a weight image. In other words, the genetic programming described above was used only by changing the target image and the weight image without manually constructing a processing program for the extraction every time the specific target is changed as in the past. The processing program can be easily constructed in the same procedure using the method as it is.

  At this time, in the formation of the processing program, for example, when learning is performed using only one learning set formed by combining the input image of FIG. 5, the target image of FIG. 10, and the weight image of FIG. 11, for example, FIG. In such a superimposed image, a phenomenon may occur in which only the person on the left side of the image is extracted. When such a phenomenon occurs, for example, even if an image as shown in FIG. 13B is input to this processing program, only the person on the left side of the image is extracted, and the pedestrian on the right side of the image cannot be extracted.

  However, as in the present embodiment, in the formation of a processing program, it is possible to avoid the occurrence of such a phenomenon by using a plurality of learning sets formed by combining an input image, a target image, and a weight image. Become. Furthermore, as shown in FIG. 18, it becomes possible to more accurately extract a person from a landscape not used in a learning set in genetic programming, and versatility to accurately extract only a person from a moving image. High processing program can be obtained.

  Further, by combining a plurality of processing programs obtained in this way into a large-scale processing program, it is possible to exhibit such effects more effectively.

  Furthermore, when a processing program is obtained by genetic programming, the search space for the solution program increases exponentially as the number of image filters (non-terminal symbols) constituting the processing program increases, and a huge search is required. Become. However, as in this embodiment, by combining a plurality of processing programs formed using different learning sets, it is possible to obtain a highly versatile processing program that can extract a specific target more easily and accurately. Become.

  In this embodiment, the purpose is to capture a front landscape from a moving vehicle and extract a pedestrian from the image. In addition, for example, a vehicle is extracted from the front landscape. It is also possible to extract a general moving object such as a vehicle or a pedestrian, or to extract a boundary between a roadway and a sidewalk. Further, by combining them, it is possible to extract the boundary between the roadway and the sidewalk and extract a vehicle or a pedestrian moving on the roadway between them.

  In the present embodiment, the output image is superimposed on the input image t for display. However, the image processing apparatus of the present embodiment can be extracted by combining the image processing apparatus of the present embodiment with another apparatus. It is also possible to send a specific target to another device for monitoring, or to measure the distance from the target with another device.

  For example, by combining the image processing device of the present embodiment and the distance measuring device, the pedestrian is identified by the image processing device of the present embodiment, and the distance from the pedestrian is measured by the distance measuring device. It is possible to issue an automatic warning at the time, and to avoid a collision by running control. Further, the distance measuring device does not need to measure the distance to the object in the entire front area of the vehicle, and the burden is reduced.

  Furthermore, the image processing apparatus according to the present embodiment can be mounted not only on a vehicle but also on, for example, an artificial intelligent robot or the like. For example, a robot can observe other environments while observing an environment with a mounted camera. Can be used to find, measure, and determine the robot's behavior for it.

It is a block diagram which shows the structure of the image processing apparatus of this embodiment. It is a figure explaining the structure of the processing program of this embodiment. It is a list of the image filter used by this embodiment. It is a figure explaining the selection method of the input image input into a processing program. It is a figure explaining the input image for 4 continuous frames, (A) shows the input image t, (B) shows the input image t-3. It is the figure explaining the output image based on the input image of FIG. FIG. 7 is a diagram showing a state in which the input image of FIG. 5A and the output image of FIG. It is a figure which shows the example of the mask filter used by this embodiment. It is a block diagram which shows the structure of the processing program formation part of this embodiment. It is a figure explaining the target image used with a fitness evaluation means. It is a figure explaining the weight image used by a fitness evaluation means. It is a figure explaining three learning sets used by this embodiment. It is a figure which shows the input image used by each learning set of FIG. 12, (A) shows a 2nd learning set, (B) shows the input image used for a 3rd learning set. It is a figure explaining the crossing of the processing program in a crossing means. It is a figure explaining the large-scale processing program formed combining a processing program. It is a figure explaining the large-scale processing program which divides | segments and displays an image using a mask filter. It is a figure which shows an example of the processing program formed by the genetic programming. It is a figure which shows the state which extracted the person from the input image which was not used for the learning set. It is a figure explaining the structure of the conventional processing program.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 21 Imaging apparatus 3 Image processing part 4 Display part 6 Processing program formation part

Claims (14)

In an image processing device that performs image processing on an image captured by an imaging device and extracts a specific target from the image,
An image processing unit for performing image processing on a plurality of types of images captured by the imaging device based on a processing program in which image filters are combined in a tree structure, and forming an output image from which the specific target is extracted An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the plurality of types of images are a plurality of images captured at time intervals by the imaging apparatus.   The image processing apparatus according to claim 1, wherein the plurality of types of images are inter-frame difference images or edge extraction images of a plurality of images captured at time intervals by the imaging apparatus.   A processing program forming unit for forming the processing program, wherein the processing program forming unit is configured to form a processing program by genetic programming using the plurality of types of images, target images, and weight images; The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The weighted image is set so that a ratio between a weight of the extraction region and a weight of the non-extraction region is a reciprocal ratio of an area ratio of the extraction region and the non-extraction region. 5. The image processing apparatus according to 4.   The image processing apparatus according to claim 4, wherein the processing program forming unit forms a processing program using a plurality of learning sets including the plurality of types of images, target images, and weight images. .   7. The fitness value used for genetic programming in the processing program forming unit is calculated to take a smaller value as the number of nodes in the processing program is larger. The image processing apparatus according to one item.   The image processing apparatus according to claim 7, wherein the ratio of the number of nodes to the fitness is configured to change according to the number of generations of evolutionary processes in genetic programming.   5. The fitness used for genetic programming in the processing program forming unit is calculated to take a larger value as the number of nodes of the two-input image filter in the processing program is larger. Item 9. The image processing device according to any one of Items 8 to 9.   The image processing apparatus according to claim 9, wherein the ratio of the number of nodes of the two-input image filter to the fitness is changed according to the number of generations of the evolution process in genetic programming.   The image processing apparatus according to any one of claims 1 to 10, wherein the processing program is formed by combining a plurality of processing programs.   The image processing apparatus according to claim 11, wherein an output image is formed by nonlinear superposition of processing by the plurality of processing programs.   The image processing apparatus according to claim 1, wherein the image filter includes a mask filter.   A display unit for displaying an image is provided, and an output image formed based on the processing program is configured to be superimposed on the input image displayed on the display unit. The image processing apparatus according to any one of claims 1 to 13.

Images