JP7090899B2 - Image processing equipment and image processing programs that use encoded illumination patterns - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing program suitable for processing a plurality of images having parallax, which are used in the fields of spectroscopic analysis and three-dimensional measurement.

Conventionally, in the field of spectroscopic analysis and the field of three-dimensional measurement, a plurality of images (including spectroscopic images) having a parallax are acquired for an analysis target or a subject to be measured, and the corresponding pixels are aligned between the images. As a result, multi-spectral data and three-dimensional shape data are acquired.

For example, Japanese Patent Application Laid-Open No. 2016-138789 describes inspection by synchronously acquiring a plurality of spectral images having different wavelengths from an optical module and analyzing the multispectral data obtained by matching each spectral image. A spectroscopic imaging system for inspecting foreign matter and the like mixed in an object has been proposed (Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-138789

However, in the stereo matching method, which is also used in Patent Document 1 and has been conventionally known as the above-mentioned alignment method, when there is no conspicuous pattern on the surface of the subject, it corresponds between a plurality of captured images. There is a problem that it is not possible to find the pixel to be used.

In order to solve such a problem, a technique of searching for a corresponding pixel by projecting a dot pattern or the like on a subject without a pattern has also been proposed. However, since there is a portion that is not illuminated at all in the conventional dot pattern or the like, there is a problem that a portion in which spectral data cannot be acquired occurs when a spectroscopic image is captured.

The present invention has been made to solve such a problem, and even if the subject captured in a plurality of images having a difference in image is unpatterned, the corresponding pixels are aligned between the images. At the same time, it is an object of the present invention to provide an image processing apparatus and an image processing program capable of acquiring spectral data in a spectral image without omission.

The image processing apparatus according to the present invention aligns the corresponding pixels between the images and acquires the spectral data in the spectral image without omission even if the subject captured in the plurality of images having the difference in image is unpatterned. In order to solve the problem of An image processing device that executes alignment processing of each pixel in each image based on the illumination pattern, wherein the illumination pattern includes an illumination region having a plurality of blocks composed of pixel groups having the same brightness value. The brightness value of each block is set to a random value other than 0 (black) while having the entire surface.

Further, as one aspect of the present invention, in order to prevent the exhaustion of random luminance values set in each block and to solve the problem of clarifying the illumination pattern and suppressing recognition errors, the illumination pattern is predetermined. A plurality of the illuminated areas of the shape are arranged, and the brightness value of each block constituting each of these illuminated areas may be set to a random value so that the illuminated areas of the same pattern do not exist.

Further, as one aspect of the present invention, in order to solve the problem of improving the efficiency and speed of the matching process of each image, the illumination pattern has a plurality of vertical grid lines and a plurality of horizontal grid lines, which are the illumination regions. It is composed of a grid line and a plurality of rectangular grids surrounded by the vertical grid line and the horizontal grid line, and at least the brightness value of each block constituting the vertical grid line and the horizontal grid line is set. It may be set to a random value so that the vertical grid line of the same pattern and the horizontal grid line of the same pattern do not exist.

Further, as one aspect of the present invention, in order to solve the problem of extracting vertical grid lines and horizontal grid lines and acquiring data as an optical cutting method, the vertical grid lines and the horizontal grid lines are configured. The range of the brightness value of the blocks and the range of the brightness value of each block constituting the grid are set to one and the other of a range smaller than a predetermined threshold value and a range larger than a predetermined threshold value, and each image captured by the image pickup apparatus. The binarization process based on the threshold value may be executed on the subject.

Further, as one aspect of the present invention, even if the image has a low brightness and a small SN ratio (Signal-Noise ratio), in order to solve the problem of positioning with high accuracy, each image imaged by the image pickup device is performed. After performing the differential processing on the image, the normalization processing for adjusting the scale of the differential value may be executed for each of the illumination regions.

Further, as one aspect of the present invention, it is possible to inspect subjects of various shapes at the same inspection speed as a conventional food foreign matter inspection device for inspecting foreign matter mixed in food or the like, and the foreign matter is automatically inspected with high accuracy and high speed. In order to solve the problem of inspection, the image pickup device is a multi-lens spectroscopic imaging camera that simultaneously captures a plurality of spectroscopic images, and the image processing device aligns and processes each pixel in the spectroscopic image. By analyzing the obtained multispectral data, foreign matter mixed in the subject may be detected.

Further, as one aspect of the present invention, in order to solve the problem of reliably acquiring accurate multispectral data and three-dimensional shape data, when there is an illumination region in which the alignment process has failed, the following (a) , (B) may be an image processing apparatus that executes at least one of the processes.
(A) Interpolation processing for interpolating and aligning based on another lighting area where the alignment processing is successful or the lighting area having the maximum matching degree.
(B) An illumination pattern different from the illumination pattern is transmitted to the projection device, and a plurality of images simultaneously acquired for the subject on which the different illumination pattern is projected are acquired from one or a plurality of image pickup devices, and the different illuminations are obtained. Retry processing that re-aligns each pixel in each image based on the pattern.

The image processing program according to the present invention aligns the corresponding pixels between the images and acquires the spectral data in the spectral image without omission even if the subject captured in the plurality of images having the difference in image is unpatterned. In order to solve the problem of An image processing program that causes a computer to execute a step of acquiring and a step of executing an alignment process of each pixel in each image based on the illumination pattern. The illumination pattern is from a group of pixels having the same brightness value. It has an illumination area having a plurality of blocks, and the brightness value of each block is set to a random value other than 0 (black).

According to the present invention, even if the subject captured in a plurality of images with parallax has no pattern, the corresponding pixels can be aligned between the images and the spectral data in the spectroscopic image can be acquired without omission. can.

It is a block diagram which shows one Embodiment of the image processing apparatus and the image processing program which concerns on this invention. It is the whole view which shows the lighting pattern of this embodiment, and is the enlarged view of a part thereof. It is an enlarged view which shows the relationship between the illumination area and a block of this embodiment. It is a figure which shows the matching process of the grid in this embodiment. In this embodiment, it is a figure which shows an example of the illumination pattern which was set so that only a vertical grid line and a horizontal grid line are unique. In this embodiment, it is a figure which shows an example of the illumination pattern in which the vertical grid line and the horizontal grid line are not provided, and each grid is set to be unique. It is a figure which shows the example which differential-processed the low-luminance illumination pattern in this embodiment. In the present embodiment, when (a) the alignment process of a part of the illumination area (grid) in the entire illumination pattern fails, (b) a part of a certain illumination area (grid) is missing and the alignment process is performed. It is a figure which shows the case which failed. It is a flowchart which shows the process executed by the image processing apparatus and the image processing program of this embodiment. It is a figure which shows the matching operation of each illumination area performed at the time of evaluating the alignment performance in this Example 1. FIG. In the first embodiment, it is a graph which shows the evaluation result of the alignment performance in the case where (a) the differentiation process and the normalization process are not executed, and (b) each process is executed.

Hereinafter, an embodiment of the image processing apparatus and the image processing program according to the present invention will be described with reference to the drawings.

As shown in FIG. 1, the image processing device 1 of the present embodiment transmits a lighting pattern for projecting to a subject to the projection device 2, and simultaneously captures a plurality of images of the subject on which the lighting pattern is projected. Is for executing the alignment processing of each pixel in each image based on the illumination pattern. Hereinafter, each configuration will be described. In the present invention, the image is a concept including not only a general color image and a gray scale image but also a spectral image.

The projection device 2 is for projecting a coded illumination pattern onto a subject. Specifically, the projection device 2 is composed of a DLP (Digital Light Processing) projector, a liquid crystal projector, an LCOS (Liquid Crystal On Silicon) projector, and the like. Then, when the illumination pattern is sent from the image processing apparatus 1, the projection device 2 projects the illumination pattern toward the subject.

The image pickup apparatus 3 captures a subject on which an illumination pattern is projected. Specifically, the image pickup apparatus 3 is composed of a multi-eye spectroscopic imaging camera that simultaneously captures a plurality of spectroscopic images, a monocular camera that captures a single image, and the like. When the image pickup device 3 is a multi-eye spectroscopic imaging camera, a plurality of spectroscopic images simultaneously captured by one image pickup device 3 are transmitted to the image processing device 1. When the image pickup device 3 is a monocular camera, a plurality of images simultaneously captured by the plurality of image pickup devices 3 are transmitted to the image processing device 1. In either case, a plurality of images with parallax are simultaneously acquired by the snapshot method.

The image processing device 1 is composed of a computer such as a personal computer, and as shown in FIG. 1, mainly performs the image processing program 1a of the present embodiment, the storage means 4 for storing various data, and various arithmetic processes. It has an arithmetic processing means 5 that is executed and functions as each component described later. Hereinafter, each configuration means will be described in detail.

The storage means 4 is composed of a hard disk, a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and is a working area for storing various data and for the arithmetic processing means 5 to execute various arithmetic processes. It functions as. In the present embodiment, as shown in FIG. 1, the storage means 4 has a program storage unit 41, a lighting pattern storage unit 42, and an image storage unit 43.

The image processing program 1a of the present embodiment is installed in the program storage unit 41. Then, the arithmetic processing means 5 executes the image processing program 1a to make the computer as the image processing device 1 function as each component described later.

The illumination pattern storage unit 42 is an area for storing an illumination pattern for projecting onto a subject. In the present embodiment, as shown in FIG. 2, the illumination pattern is composed of a plurality of vertical grid lines, a plurality of horizontal grid lines, and a plurality of rectangular grids surrounded by these vertical grid lines and horizontal grid lines. It is configured.

Further, in the present embodiment, each of the vertical grid line, the horizontal grid line, and the grid is a lighting area having a plurality of blocks composed of pixel groups having the same luminance value, as shown in FIG. 3, and this lighting area. Has on the entire surface of the lighting pattern. Then, the luminance value of each block constituting each of the vertical grid line, the horizontal grid line, and the grid is 0 (black) so that the vertical grid line of the same pattern, the horizontal grid line of the same pattern, and the grid of the same pattern do not exist. ) Is set to a random value. For example, in the case of a gray scale of 256 steps from 0 to 255, it is randomly encoded by any value from 1 to 255.

According to the illumination pattern encoded as described above, since a predetermined pattern is projected on the surface of the subject, it is possible to perform the alignment processing of the corresponding pixels between a plurality of images having parallax. Further, since the projection device 2 illuminates the subject on the entire surface of the illumination pattern and functions as illumination, there is no portion that is not exposed to light. Therefore, when an image is taken with a spectroscopic imaging camera or the like, the spectral data is acquired without omission.

Further, since the illumination area is composed of blocks composed of pixels having the same luminance value, the random luminance values set in each block are not exhausted. Further, as compared with the case where the brightness value is set for each pixel, the illumination pattern becomes clear and the recognition failure is suppressed.

Further, according to the illumination pattern of the present embodiment, the relative positional relationship between the vertical grid line, the horizontal grid line, and the grid becomes known. Therefore, for example, as shown in FIG. 4A, when the grids A and B are matched, if the grid A can be matched, the corresponding grid of the grid B is based on the relative positional relationship with the grid A. To identify. Then, as shown in FIG. 4B, only the grid existing around the corresponding grid is limited as the search range of the grid B, so that the matching process of each image is made more efficient and faster.

However, the illumination pattern is not limited to the above-mentioned configuration, and is appropriately changed as long as it has an illumination area on the entire surface and the brightness value of each block is set to a random value other than 0 (black). can do. For example, from the viewpoint of improving the efficiency and speed of the matching process described above, it is sufficient that the relative positional relationship between the grid lines or the grids is known. Therefore, at least the luminance value of each block constituting the vertical grid line and the horizontal grid line may be set to a random value so that the vertical grid line of the same pattern and the horizontal grid line of the same pattern do not exist. In this case, even if each grid has the same pattern, it can be uniquely defined by the four line segments constituting the vertical grid line and the horizontal grid line. For example, as shown in FIG. 5, the grid X can be defined by four line segments (horizontal 1, horizontal 2, vertical 1, vertical 2) surrounding the grid X.

Alternatively, as shown in FIG. 6, vertical grid lines and horizontal grid lines are not provided, and a plurality of grids are arranged without gaps so that the luminance values of the blocks constituting each grid do not have the same pattern. It may be set to a random value. This is because the relative positional relationship between the grids becomes known even with this configuration.

Further, in the present embodiment, it is assumed that the image pickup device 3 provided with a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) in which photodiodes are arranged in the horizontal direction and the vertical direction is used as the image pickup element. For this reason, by using an illumination pattern consisting of only vertical or horizontal straight lines, the occurrence of geometric distortion is suppressed, and the matching process is simplified and speeded up. However, if the influence of geometric distortion can be ignored, the illumination pattern is not limited to the grid pattern, and a plurality of illumination regions having a predetermined shape including diagonal lines, curves, etc. are arranged. , The brightness value of each block constituting each of these illumination areas may be set to a random value so that the illumination areas of the same pattern do not exist.

Further, if it is possible to guarantee the speeding up of the matching process described above by a high-spec computer or the like, the lighting pattern is not divided into a plurality of lighting areas, and blocks of random brightness values are formed without gaps on the entire surface of the lighting pattern. It may be arranged. That is, the illumination pattern according to the present invention has an illumination region having a plurality of blocks consisting of at least a group of pixels having the same luminance value on the entire surface, and the luminance value of each block is random other than 0 (black). It suffices if it is set to a value.

In the present embodiment, the range of the luminance value of each block constituting the vertical grid line and the horizontal grid line and the range of the luminance value of each block constituting the grid are one of a range smaller than a predetermined threshold value and a range larger than the predetermined threshold value. And the other. Therefore, when the binarization process based on the threshold value is executed on the image in which the illumination pattern of the present embodiment is captured, the vertical grid lines and the horizontal grid lines are extracted, so that the data as the optical cutting method is acquired. It becomes possible.

Further, in the present embodiment, the illumination pattern storage unit 42 stores a plurality of types of illumination patterns having different patterns in order to execute the retry process described later.

The image storage unit 43 is an area for storing an image captured by the image pickup device 3. In the present embodiment, the image storage unit 43 stores a plurality of images simultaneously acquired by the image acquisition unit described later as one image set.

The arithmetic processing means 5 is composed of a CPU (Central Processing Unit) or the like, and by executing the image processing program 1a installed in the program storage unit 41, the illumination pattern transmission unit 51 and the illumination pattern transmission unit 51 are executed, as shown in FIG. , Image acquisition unit 52, differentiation processing unit 53, normalization processing unit 54, binarization processing unit 55, alignment processing unit 56, interpolation processing unit 57, and foreign matter detection processing unit 58. It has become. Hereinafter, each component will be described in more detail.

The illumination pattern transmission unit 51 transmits the illumination pattern to the projection device 2. In the present embodiment, the illumination pattern transmission unit 51 reads out the illumination pattern stored in the illumination pattern storage unit 42 and transmits the illumination pattern to the projection device 2. In this embodiment, the lighting pattern once created can be reused for all shooting. Therefore, it is not necessary to create and transmit a new lighting pattern every time an image is taken.

The image acquisition unit 52 acquires an image from the image pickup device 3. In the present embodiment, the image acquisition unit 52 causes one or a plurality of image pickup devices 3 to simultaneously capture a plurality of images by a snapshot method at a timing when the subject is within the angle of view of the image pickup device 3. Then, a plurality of images are acquired from those image pickup devices 3 and stored in the image storage unit 43.

The differential processing unit 53 executes differential processing on the image. In the present embodiment, the differential processing unit 53 has a differential processing function using a differential filter such as a Sobel filter. Then, when the captured image has a portion having a small luminance value, the differential processing unit 53 performs differential processing in both the vertical direction and the horizontal direction to emphasize the illumination pattern.

The normalization processing unit 54 executes normalization processing for adjusting the scale of the differential value for each illumination region on the differentiated image. In the present embodiment, the normalization processing unit 54 specifies the maximum value and the minimum value of the differential value of each pixel for each illumination region (vertical grid line, horizontal grid line, and grid) which is a unit for matching processing, and these are specified. Based on the maximum and minimum values, the scale of the derivative value is adjusted between a plurality of vertical grid lines, a plurality of horizontal grid lines, and a plurality of grids.

For example, as shown in FIG. 7, a low-luminance illumination pattern having a luminance value of 25% with respect to the original pattern is difficult to see, but by executing differential processing and normalization processing, the illumination pattern is emphasized and is easy to see. Become. As described above, in the present embodiment, after the differential processing is executed for the low-luminance image, the normalization processing for adjusting the scale of the differential value is executed for each illumination region. This makes it possible to align even an image with low brightness and a small SN ratio with high accuracy.

The binarization processing unit 55 executes binarization processing according to a predetermined threshold value for each image captured by the image pickup apparatus 3. In the illumination pattern of the present embodiment, the range of the luminance value of each block constituting the vertical grid line and the horizontal grid line and the range of the luminance value of each block constituting the grid are a range smaller than a predetermined threshold value and a range larger than the predetermined threshold value. It is set to one and the other. Therefore, when the binarization processing unit 55 executes the binarization processing according to the threshold value, the vertical grid lines and the horizontal grid lines are extracted, and the data as the optical cutting method is acquired.

The alignment processing unit 56 executes the alignment processing of each pixel in each image based on the illumination pattern. In the present embodiment, the alignment processing unit 56 searches for corresponding pixels in a plurality of images having parallax by using an algorithm such as a stereo matching method, and aligns the pixels by correcting the deviation between the pixels. It has become like. As a result, in the case of a spectroscopic image, accurate multispectral data can be obtained, and in the case of an image for three-dimensional measurement, accurate three-dimensional shape data can be obtained.

The interpolation processing unit 57 executes the interpolation processing when there is a lighting area in which the alignment processing has failed. In the present embodiment, for example, as shown in FIG. 8A, when the interpolation processing unit 57 fails in the alignment processing of a part of the illumination pattern (grid (6, G)) in the entire illumination pattern. , Other lighting areas existing around the lighting area (grid (5, F) to (5, H), (6, F), (6, H), (7, F) to (7, H) ) Check if the alignment process is successful. Then, if successful, since the positional relationship of each of these uniquely identifiable grids is known, the interpolation processing unit 57 interpolates the positions of the grids (6, G) based on the surrounding grids.

Further, in the present embodiment, as shown in FIG. 8B, when a part of a certain lighting area (grid) is missing and the alignment processing fails, the interpolation processing unit 57 has a random pattern of each lighting area. Since there is only one, there is a matching part in the illumination area, and the illumination area with the maximum degree of matching is specified. Then, since the positional deviation between the images is calculated by aligning with the illumination area, the interpolation processing unit 57 interpolates the positional deviation amount of the missing portion based on the peripheral positional deviation amount. It has become.

Further, in the present embodiment, when there is a lighting area where the alignment process has failed and the interpolation process also fails, or the retry process may be executed instead of the interpolation process. In this retry process, an illumination pattern different from the initially illuminated illumination pattern is transmitted to the projection device 2, and a plurality of images simultaneously captured for the subject on which the different illumination pattern is projected are captured by one or a plurality of image pickup devices 3. It is a process of re-aligning each pixel in each image based on the different illumination patterns obtained from the above.

That is, in the present embodiment, when there is a lighting area in which the alignment process has failed, at least one of the following processes (a) and (b) is executed.
(A) Interpolation processing for interpolating and aligning based on another lighting area where the alignment processing is successful or the lighting area having the maximum matching degree.
(B) An illumination pattern different from the illumination pattern for which the alignment process has failed is transmitted to the projection device 2, and a plurality of images simultaneously acquired for a subject on which the different illumination patterns are projected are transmitted from one or a plurality of image pickup devices 3. Retry processing that acquires and realigns each pixel in each image based on different illumination patterns.

The foreign matter detection processing unit 58 detects foreign matter such as human hair mixed in a subject such as food based on the multispectral data obtained from the spectroscopic image. In the present embodiment, the foreign matter detection processing unit 58 executes analysis such as principal component analysis on the multispectral data of the subject, and acquires the distribution of the feature amount of the subject and the feature amount of the foreign matter. When a feature amount having a distribution different from that of the subject is detected, the foreign matter detection processing unit 58 detects that the foreign matter is mixed in the subject.

Next, the operations of the image processing device 1 and the image processing program 1a of the present embodiment will be described with reference to the flowchart of FIG. In the following description, an example of capturing an image (spectral image) of a subject conveyed by a conveyor and detecting foreign matter mixed in the subject based on the image will be described.

When using the image processing device 1 and the image processing program 1a of the present embodiment, first, as a preliminary preparation, a lighting pattern created in advance is stored in the lighting pattern storage unit 42. In this embodiment, a plurality of types of lighting patterns having different patterns are stored in the lighting pattern storage unit 42 so that the retry process can be executed.

Next, when the illumination pattern transmission unit 51 transmits the illumination pattern to the projection device 2 (step S1), the illumination pattern is projected from the projection device 2 at a predetermined position on the conveyor. Then, when the image pickup apparatus 3 simultaneously captures a plurality of images of the subject at the timing when the subject passes through the predetermined position, the image acquisition unit 52 acquires each image (step S2).

As a result, a plurality of images in which the illumination pattern is projected on the surface of the subject can be obtained, so that the corresponding pixels can be aligned between the plurality of images having parallax even if the subject has no pattern. It becomes. Further, since the illumination area provided on the entire surface of the illumination pattern is composed of blocks having a luminance value other than 0 (black), the entire subject is illuminated. Therefore, when the spectroscopic image is taken, the spectral data is acquired without omission. Further, in the present embodiment, since the spectral image of the subject is acquired by one imaging by the snapshot method, the inspection speed is increased when inspecting a large number of subjects.

Subsequently, in the present embodiment, when the image acquired in step S2 has a portion having a small luminance value (step S3: YES), after the differential processing unit 53 executes the differential processing for each image (step S4). ), The normalization processing unit 54 executes the normalization processing for each illumination area (step S5), and adjusts the scale of the differential value. As a result, even if the image has low brightness and a small SN ratio, the illumination pattern is emphasized and it is possible to align the image with high accuracy.

Next, after the execution of step S5 or when there is no portion having a small luminance value in each image acquired in step S2 (step S3: NO), in the present embodiment, it is necessary to acquire data as an optical cutting method. Judge whether or not. Then, when the data is required (step S6: YES), the binarization processing unit 55 executes the binarization processing for each image (step S7). As a result, vertical grid lines and horizontal grid lines are extracted from the illumination pattern of the present embodiment, so even if acquisition of multispectral data or three-dimensional shape data fails, data as an optical cutting method can be obtained. To be acquired.

After the execution of step S7, or when the data of the optical cutting method is unnecessary (step S6: NO), the alignment processing unit 56 executes the alignment processing of each pixel in each image based on the illumination pattern (step S8). ). As a result, accurate multispectral data can be obtained when a spectroscopic image is taken. Further, when an image for three-dimensional measurement is captured, accurate three-dimensional shape data can be obtained.

Subsequently, in the present embodiment, when there is an illuminated area in which the alignment process in step S8 has failed (step S9: YES), the interpolation processing unit 57 executes the interpolation process or the retry process (step S9: YES). S10). According to the interpolation processing, even if the alignment processing of a part of the lighting area in the entire lighting pattern fails, the interpolation processing unit 57 interpolates the position of the lighting area. Further, even if a part of a certain illumination area is missing and the alignment processing fails, the interpolation processing unit 57 interpolates the amount of misalignment of the missing portion. Therefore, even a subject having fine irregularities on the surface and difficult to align can be matched with high accuracy.

Further, according to the retry process, the process from step S1 to step S9 described above is repeated using an illumination pattern different from the illumination pattern initially irradiated. Therefore, even if the interpolation processing fails, retries are performed until the alignment processing of each pixel is successful, and multispectral data and three-dimensional shape data are acquired more reliably.

When the alignment process in step S8 is successful (step S9: NO), the foreign matter detection processing unit 58 detects the foreign matter mixed in the subject based on the multispectral data obtained from the spectroscopic image (step S11). This makes it possible to inspect subjects of various shapes at an inspection speed equivalent to that of X-ray foreign matter inspection devices and metal detectors that have already been introduced in food factories and the like. Therefore, it is possible to automatically inspect foreign substances such as human hair, plastic pieces, and insects, which have been visually inspected, with high accuracy and high speed.

According to the image processing device 1 and the image processing program 1a according to the present invention as described above, the following effects are obtained.
1. 1. Even if the subject captured in a plurality of images with parallax has no pattern, the corresponding pixels can be aligned between the images, and the spectral data in the spectroscopic image can be acquired without omission.
2. 2. It is possible to prevent the exhaustion of random brightness values set in each block, clarify the lighting pattern, and suppress recognition errors.
3. 3. The matching process of each image can be made more efficient and faster.
4. It is possible to extract vertical grid lines and horizontal grid lines and acquire data as an optical cutting method.
5. Even an image with low brightness and a small SN ratio can be aligned with high accuracy.
6. Subjects of various shapes can be inspected at the same inspection speed as conventional inspection equipment, and foreign matter can be automatically inspected with high accuracy and high speed.
7. Accurate multispectral data and 3D shape data can be reliably acquired.

Next, specific examples of the image processing apparatus 1 and the image processing program 1a according to the present invention will be described.

In the first embodiment, a simulation experiment was conducted to theoretically evaluate the alignment performance executed by the image processing device 1 and the image processing program 1a of the present embodiment.

As the experimental conditions, a flat subject with no pattern on the surface was set as the subject. As the illumination pattern, as shown in FIG. 2, an 8-bit gradation pattern that is composed of 15 vertical grid lines, 13 horizontal grid lines, and 168 grids and can be output by a projector is used. did. Then, two images of lighting patterns having the same average brightness value (1 to 99%) are prepared, and patterns (vertical grid lines, horizontal grid lines, grid lines) for each lighting area constituting each lighting pattern are prepared. ) Matches.

As the evaluation items, "whether or not the patterns of the illumination area completely match" and "evaluation value" at that time were calculated using the average luminance value as a parameter. For matching of each illumination area, the normalized difference squared sum was used as a template matching method. This normalized difference squared sum has the highest degree of matching when the value becomes the minimum (0), so the value obtained by subtracting the normalized difference squared sum from 1 is used as the evaluation value, and the evaluation value when a perfect match is made is used. The maximum is (1). That is, the evaluation values are 0 if they do not match, and are greater than 0 and 1 or less if they match. The fact that they do not match means that the originally unique pattern of the lighting area is indistinguishable from the patterns of other lighting areas.

Under the above conditions, as shown in FIG. 10, vertical grid lines are scanned horizontally, horizontal grid lines are scanned vertically, grids are raster-scanned, template matching is performed, and evaluation at each luminance value is performed. The value was calculated. Further, separately, the evaluation value when the above-mentioned differentiation processing and normalization processing was performed on each lighting pattern was calculated. The result is shown in FIG. In FIG. 11, when the alignment was possible, the evaluation value was plotted, and when the alignment could not be achieved, the evaluation value was plotted as 0.

As shown in FIG. 11A, it was confirmed that the evaluation value increased when the luminance value reached about 52 to 53% and the alignment could be performed even when the differentiation processing and the normalization processing were not executed. Looking at the evaluation values for each illumination area, it can be seen that the horizontal grid lines are the easiest to align and the grid is the most difficult to align. On the other hand, as shown in FIG. 11B, it was confirmed that when the differentiation process and the normalization process were executed, almost the alignment could be performed in any of the illumination regions if the luminance value was 3% or more.

According to the above-mentioned first embodiment, it is shown that the illumination pattern according to the present invention can align the corresponding pixels among a plurality of images even if the subject has no pattern. It was also shown that the alignment performance is improved by performing the differential processing and the normalization processing.

The image processing device 1 and the image processing program 1a according to the present invention are not limited to the above-described embodiments, and can be appropriately modified.

For example, in the above-described embodiment, when the captured image has low brightness and the SN ratio is small, the differential processing and the normalization processing are executed, but in an environment where a high-brightness image can always be captured, it is not always necessary. You don't have to do it. Further, in the above-described embodiment, when the alignment process fails, the interpolation process and the retry process are executed, but it is not always necessary to execute the process, and the failed subject may be visually inspected.

The image processing apparatus 1 and the image processing program 1a according to the present invention are suitable for use in the fields of spectroscopic analysis and three-dimensional measurement. For example, in the field of spectroscopic analysis, it may be used as a foreign matter detecting device for detecting a foreign matter mixed in food or the like based on multispectral data. Further, in the field of three-dimensional measurement, there is a possibility of using it as a three-dimensional shape measuring device for obtaining a three-dimensional shape of an object to be grasped by a robot hand or the like.

1 Image processing device 1a Image processing program 2 Projection device 3 Image pickup device 4 Storage means 5 Arithmetic processing means 41 Program storage unit 42 Lighting pattern storage unit 43 Image storage unit 51 Lighting pattern transmission unit 52 Image acquisition unit 53 Differentiation processing unit 54 Normalization Processing unit 55 Binarization processing unit 56 Alignment processing unit 57 Interpretation processing unit 58 Foreign matter detection processing unit

Claims (8)

A lighting pattern for projecting to a subject is transmitted to a projection device, and a plurality of images simultaneously captured for the subject on which the lighting pattern is projected are acquired from one or a plurality of image pickup devices, and each of them is based on the lighting pattern. An image processing device that executes alignment processing for each pixel in an image.
The illumination pattern has an illumination region having a plurality of blocks composed of pixels having the same luminance value on the entire surface, and the luminance value of each block is set to a random value other than 0 (black). The image processing device. The illumination pattern is formed by arranging a plurality of the illumination regions having a predetermined shape, and the brightness value of each block constituting each of these illumination regions is set to a random value so that the illumination regions of the same pattern do not exist. , The image processing apparatus according to claim 1. The illumination pattern is composed of a plurality of vertical grid lines, a plurality of horizontal grid lines, and a plurality of rectangular grids surrounded by the vertical grid lines and the horizontal grid lines, which are the illumination areas. ,
At least the luminance values of the vertical grid lines and the blocks constituting the horizontal grid lines are set to random values so that the vertical grid lines of the same pattern and the horizontal grid lines of the same pattern do not exist. The image processing apparatus according to claim 1 or 2. The range of the luminance value of each block constituting the vertical grid line and the horizontal grid line and the range of the luminance value of each block constituting the grid are set to one of a range smaller than a predetermined threshold value and a range larger than the predetermined threshold value, and the other. The image processing apparatus according to claim 3, wherein each image captured by the image pickup apparatus is subjected to binarization processing according to the threshold value. One of claims 1 to 4, wherein a differential process is executed for each image captured by the image pickup device, and then a normalization process for adjusting the scale of the differential value is executed for each illumination region. The image processing device described. The image pickup device is a multi-eye spectroscopic imaging camera that simultaneously captures a plurality of spectroscopic images, and the image processing device analyzes multispectral data obtained by aligning each pixel in the spectroscopic image. The image processing apparatus according to any one of claims 1 to 5, which detects foreign matter mixed in the subject. The image processing according to any one of claims 1 to 6, wherein at least one of the following processes (a) and (b) is executed when there is an illuminated area in which the alignment process has failed. Device;
(A) Interpolation processing for interpolating and aligning based on another lighting area where the alignment processing is successful or the lighting area having the maximum matching degree;
(B) An illumination pattern different from the illumination pattern is transmitted to the projection device, and a plurality of images simultaneously acquired for the subject on which the different illumination pattern is projected are acquired from one or a plurality of image pickup devices, and the different illuminations are obtained. Retry processing that re-aligns each pixel in each image based on the pattern. The step of transmitting the illumination pattern for projecting to the subject to the projection device,
A step of acquiring a plurality of images simultaneously captured for a subject on which the illumination pattern is projected from one or a plurality of imaging devices, and a step of acquiring the plurality of images.
A step of executing the alignment process of each pixel in each image based on the illumination pattern, and
Is an image processing program that causes a computer to execute
The illumination pattern has an illumination region having a plurality of blocks composed of pixels having the same luminance value on the entire surface, and the luminance value of each block is set to a random value other than 0 (black). The image processing program.

Images