JP7143649B2 - Image processing device, image processing system, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an image processing system, an image processing method, and a program.

2. Description of the Related Art There are surveillance systems that detect anomalies and the like by analyzing images captured by imaging devices such as video cameras. In such a system, a method of comparing a reference image, which serves as a criterion for determination, with a detected image obtained by photographing a specimen to be determined is used. For example, there is known a method of making a correct/incorrect determination based on a comparison result between a histogram of pixel values of a reference image and a histogram of pixel values of a detection image.

For example, in an automatic inspection apparatus that selects foreign matter candidates by comparing an inspection image with a threshold value, each pixel of the inspection image is are aggregated for each luminance, the gradation range of the histogram is expanded, the image size of the expanded inspection image is reduced to generate a reduced image, and the inspection image is specified at a position corresponding to the pixel of the reduced image. A method of setting the luminance of each pixel in a reduced image to the average value of the luminance of a plurality of pixels adjacent to a pixel and a specific pixel is disclosed (Patent Document 1).

When the correctness judgment is performed based on the result of comparison between the histogram of the reference image and the histogram of the detected image, a slight deviation in the bins in which the frequency values are voted may result in abnormalities ( mismatch) may be determined. For example, if a comparison is made using a histogram with 256 bins corresponding to 256 different color components, the votes in the histogram of the reference image are concentrated in the 200th bin, and the votes in the histogram of the detection image are in the 200th bin. Consider the case of concentrating on the 201 bin. In such a case, since the difference in hue between two adjacent bins is slight, it can be determined that the hues of the reference image and the detected image substantially match. Such slight bin shifts can occur depending on factors other than the analyte, such as changes in ambient light, placement of the analyte within the sensing region, and the like. Depending on the conventional technology, it is not possible to sufficiently avoid such unjust right/wrong determinations due to factors other than the sample, and it is not possible to perform right/wrong determinations flexibly. Therefore, there is room for improvement in terms of the practicality of correctness determination.

The present invention has been made in view of the above, and an object of the present invention is to improve the practicality of correct/incorrect judgment.

In order to solve the above-described problems and achieve the object, an image processing apparatus according to one aspect of the present invention provides a first histogram that is a histogram of pixel values of a first image and pixel values of a second image. a histogram generating unit that generates a second histogram that is a histogram of the frequency values in the first histogram and the second histogram into one or more bins located near the bin corresponding to the frequency value A leveling unit that performs a leveling process for dispersion, a first leveled histogram that is the leveled first histogram, and a second leveled histogram that is the leveled second histogram and a comparison unit that generates comparison information indicating a comparison result.

ADVANTAGE OF THE INVENTION According to this invention, the practicality of right/wrong determination can be improved.

FIG. 1 is a diagram illustrating a hardware configuration example of an image processing system according to the first embodiment. FIG. 2 is a diagram illustrating a hardware configuration example of an information processing apparatus according to the first embodiment; FIG. 3 is a block diagram illustrating a functional configuration example of the information processing apparatus according to the first embodiment; FIG. 4 is a flowchart illustrating an example of processing by the information processing apparatus according to the first embodiment; FIG. 5 is a diagram showing an example of a state when setting an area on the setting screen according to the first embodiment. FIG. 6 is a diagram showing an example of the state of the setting screen when setting the reference image according to the first embodiment. FIG. 7 is a diagram illustrating an example of the state of the setting screen when the detection image is set according to the first embodiment. FIG. 8 is a flowchart illustrating a specific example of processing by a histogram generation unit and a smoothing unit according to the first embodiment; FIG. 9 is a diagram illustrating an example of a histogram before smoothing according to the first embodiment; FIG. 10 is a diagram illustrating an example of a histogram after smoothing according to the first embodiment; FIG. 11 is a diagram illustrating an example of a first state when setting threshold values on the setting screen according to the first embodiment. FIG. 12 is a diagram illustrating an example of a second state during threshold setting on the setting screen according to the first embodiment. 13A and 13B are diagrams illustrating examples of correct/incorrect determination results according to the first embodiment. FIG. 14 is a diagram illustrating an example of a third state during threshold setting on the setting screen according to the first embodiment. FIG. 15 is a diagram illustrating an example of the state of the setting screen when setting is completed according to the first embodiment. FIG. 16 is a diagram showing an example of the first state when setting threshold values on the setting screen according to the second embodiment. FIG. 17 is a diagram illustrating an example of a second state during threshold setting on the setting screen according to the second embodiment. FIG. 18 is a block diagram illustrating a functional configuration example of an information processing apparatus according to the third embodiment; FIG. 19 is a flowchart illustrating an example of processing by the information processing apparatus according to the third embodiment; FIG. 20 is a flowchart illustrating a specific example of processing by a recalculation unit according to the third embodiment; FIG. 21 is a diagram showing an example of the lower limit value of the S component according to the third embodiment. FIG. 22 is a diagram showing examples of the lower limit value and upper limit value of the V component according to the third embodiment. FIG. 23 is a diagram showing an example of a state in which pixels are disabled in the third embodiment. FIG. 24 is a diagram showing an example of the state of the setting screen when the reference image is set according to the third embodiment. FIG. 25 is a diagram showing an example of a state during determination processing of a setting screen according to the third embodiment. FIG. 26 is a flow chart showing a specific example of processing by the origin moving unit according to the third embodiment. FIG. 27 is a diagram showing an example of a histogram before execution of origin movement processing according to the third embodiment. FIG. 28 is a diagram showing an example of a histogram after execution of origin movement processing according to the third embodiment. FIG. 29 is a block diagram illustrating a functional configuration example of an information processing apparatus according to the fourth embodiment; FIG. 30 is a flowchart illustrating an example of processing by the information processing apparatus according to the fourth embodiment; FIG. 31 is a flowchart illustrating a specific example of processing by a color setting unit according to the fourth embodiment; FIG. 32 is a diagram showing an example of a state during color setting processing on the setting screen according to the fourth embodiment. FIG. 33 is a diagram showing an example of a state when reflecting the setting color on the setting screen according to the fourth embodiment. FIG. 34 is a diagram showing an example of a setting screen according to the fifth embodiment. FIG. 35 is a diagram showing an example of a state in which one designated point is designated on the setting screen according to the fifth embodiment. FIG. 36 is a diagram showing an example of a state in which two specified points are specified on the setting screen according to the fifth embodiment. FIG. 37 is a diagram showing an example of a state in which three designated points are designated on the setting screen according to the fifth embodiment. FIG. 38 is a diagram showing an example of a state in which four designated points are designated on the setting screen according to the fifth embodiment. FIG. 39 is a diagram showing an example of a state when deleting designated points on the setting screen according to the fifth embodiment. FIG. 40 is a diagram showing an example of the state after the specified point is deleted on the setting screen according to the fifth embodiment. FIG. 41 is a diagram showing an example of a state when adjusting the allowable error lower limit value and the allowable error upper limit value on the setting screen according to the fifth embodiment. FIG. 42 is a flowchart illustrating an example of processing by the information processing apparatus according to the fifth embodiment; FIG. 43 is a block diagram illustrating a functional configuration example of an information processing apparatus according to the sixth embodiment; FIG. 44 is a flowchart illustrating an example of processing by the information processing apparatus according to the sixth embodiment; FIG. 45 is a flowchart illustrating a specific example of processing by an extension unit according to the sixth embodiment. FIG. 46 is a diagram showing an example of a histogram before expansion according to the sixth embodiment. FIG. 47 is a diagram showing an example of a histogram after expansion according to the sixth embodiment. FIG. 48 is a diagram showing an example of a state before expansion of the setting screen according to the sixth embodiment. FIG. 49 is a diagram showing an example of a state after expansion of the setting screen according to the sixth embodiment.

Embodiments of an image processing apparatus, an image processing system, an image processing method, and a program according to the present invention will be described in detail below with reference to the drawings. The present invention is not limited by the following embodiments, and components in the following embodiments include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. . Various omissions, replacements, changes, and combinations of components can be made without departing from the gist of the following embodiments.

(First embodiment)
FIG. 1 is a diagram showing a hardware configuration example of an image processing system 1 according to the first embodiment. The image processing system 1 includes imaging devices 2 a to 2 f, an information processing device 3 (image processing device), a network 4 and an external device 5 .

The imaging devices 2a to 2f capture an image of a subject by converting light emitted (reflected) from the subject into an electrical signal, and capture a moving image (for example, 10 [FPS]) composed of a plurality of frames (still image data). A video camera that generates video data. The imaging devices 2a to 2f, for example, photograph production equipment, production lines, etc. that produce a plurality of products, and generate video data for executing correct/incorrect judgments for detecting abnormalities in workpieces (specimens) that are manufactured products. . In this embodiment, a plurality of (six in this example) imaging devices 2a to 2f are provided, but one may be provided.

The information processing device 3 is a PC (Personal Computer), a work station, or the like, which functions as a device for judging whether a work is correct or not based on the video data generated by each imaging device 2a to 2f. The information processing device 3 according to the present embodiment is connected to an external device 5 that constitutes a production facility or the like so as to enable communication according to, for example, the Fieldbus standard.

The network 4 is, for example, an Ethernet (registered trademark) standard network for connecting the imaging devices 2 a to 2 f and the information processing device 3 . In this case, data communication is performed on the network 4 by protocols such as TCP (Transmission Control Protocol)/IP (Internet Protocol). In this case, the imaging devices 2a to 2f and the information processing device 3 have MAC (Media Access Control) addresses for communication using the TCP/IP protocol, and are assigned IP addresses such as private IP addresses. A specific configuration of the network 4 is, for example, a star wiring format in which the imaging devices 2a to 2f and the information processing device 3 are connected to a switching hub having a plurality of ports by LAN (Local Area Network) cables. The network 4 is not limited to TCP/IP. For example, the information processing device 3 has a plurality of VGA (Video Graphics Array) terminals or USB (Universal Serial Bus) ports and a plurality of imaging devices. 2a to 2f may be connected to the information processing apparatus 3 by VGA cables or USB cables.

FIG. 2 is a diagram showing a hardware configuration example of the information processing device 3 according to the first embodiment. The information processing device 3 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an external storage device 14, a display 15, a network I/F 16, a keyboard 17, a mouse 18, a DVD (Digital Versatile Disc) drive 19, external device I/F 21, and speaker 22 are included.

The CPU 11 is a device comprising one or more circuits that perform arithmetic processing to control the operation of the information processing device 3 as a whole. The ROM 12 is a nonvolatile storage device that stores programs for the information processing device 3 . RAM 13 is a volatile storage device used as a work area for CPU 11 .

The external storage device 14 is a storage device such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores various data such as video data and setting information generated by the imaging devices 2a to 2f.

The display 15 is a display device for displaying a cursor, a menu, a window, characters, an image, a screen of an application for executing right/wrong determination, and the like. The display 15 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, a plasma display, an organic EL (Electroluminescence) display, or the like. The display 15 is connected to the main body of the information processing apparatus 3, for example, by a VGA cable, an HDMI (registered trademark) (High-Definition Multimedia Interface) cable, an Ethernet cable, or the like. The display 15 may have a mechanism for enabling touch panel input operations.

A network I/F 16 is an interface for connecting to the network 4 for data communication. The network I/F 16 is, for example, a NIC (Network Interface Card) that enables communication according to the TCP/IP protocol. Specifically, the information processing device 3 acquires video data from the imaging devices 2a to 2f via the network 4 and the network I/F 16. FIG.

The keyboard 17 is an input device for inputting characters, numbers, selection of various instructions, cursor movement, setting of setting information, and the like. The mouse 18 is an input device for selecting (executing) various instructions, selecting a processing target, moving a cursor, setting setting information, and the like.

The DVD drive 19 is a device that controls reading, writing, and deleting data from a DVD 20, which is an example of a removable storage medium.

The external device I/F 21 is an interface for connecting with the external device 5 for data communication. The external device I/F 21 is, for example, an interface card that enables communication according to the Fieldbus standard. Specifically, the information processing device 3 performs data communication with the external device 5 via the external device I/F 21 .

The speaker 22 is a device that outputs sound according to the operation of the application.

The above-described CPU 11, ROM 12, RAM 13, external storage device 14, display 15, network I/F 16, keyboard 17, mouse 18, DVD drive 19, external device I/F 21, and speaker 22 are connected to buses such as an address bus and a data bus. are communicatively connected to each other by The display 15 is connected to the network I/F 16 when connected by an Ethernet cable, and in this case, data communication is performed by protocols such as TCP/IP.

The hardware configuration shown in FIGS. 1 and 2 is merely an example, and the image processing system 1 and the information processing device 3 should be constructed using appropriate hardware and software according to usage conditions. be.

FIG. 3 is a block diagram showing a functional configuration example of the information processing device 3 according to the first embodiment. The information processing device 3 includes a video receiving unit 101 (acquisition unit), a storage unit 102, an input unit 103, an area setting unit 104, a reference image setting unit 105 (image setting unit), a detection image setting unit 106 (image setting unit), A histogram generation unit 107 , a smoothing unit 108 , a comparison unit 109 , a determination unit 110 , a threshold value setting unit 111 , a display control unit 112 and an external output unit 113 are included.

The video receiving unit 101 performs data communication with the imaging devices 2a to 2f via the network 4 and receives video data from the imaging devices 2a to 2f. The video receiving unit 101 causes the storage unit 102 to store the received video data.

The storage unit 102 stores various data such as video data and setting information received by the video reception unit 101 .

The input unit 103 receives input operations by the user regarding various processes executed by the information processing device 3 .

The area setting unit 104 performs processing for setting an arbitrary area from the image indicated by the image data (images captured by the imaging devices 2a to 2f). Although the method of setting the area should not be particularly limited, for example, the user operates a pointing device such as the mouse 18 on the image projected on the display device 115 such as the display 15 to designate an arbitrary area. It can be done by

A reference image setting unit 105 performs processing for setting a reference image (first image) that serves as a reference for determination based on the area set by the area setting unit 104 .

Based on the region set by the region setting unit 104, the detection image setting unit 106 performs processing for setting a detection image (second image) in which the specimen to be determined is captured.

The histogram generation unit 107 generates a reference histogram (first histogram), which is a histogram of pixel values of the reference image, and a detection histogram (second histogram), which is a histogram of pixel values of the detection image. Although pixel values for generating a histogram should not be particularly limited, for example, HSV (Hue/Saturation/Value), HLS (Hue/Saturation )·luminance (Lightness/Luminance), a value indicating a position in a color space, a value (luminance value) indicating a luminance in a gray scale, or the like. If the reference and sensed images are in color, then pixel values are used that indicate locations in a color space such as HSV. For example, when HSV or HLS are used as pixel values, the bins (classes) in the histogram are set corresponding to the number of H components (hue) (for example, 256), and the number of occurrences (frequency value) of each H component is Vote in the corresponding bin.

A leveling unit 108 performs a leveling process on the reference histogram and the detection histogram. The smoothing process is a process of distributing the frequency values in the histogram to one or more bins located near the bin corresponding to the frequency values. Although the method of distributing the frequency values is not particularly limited, for example, the frequency values of the 100th bin are distributed to the 99th and 101st bins adjacent to the 100th bin. It may also be distributed to bins that are not directly adjacent to the 100th bin but that are located near it (eg, the 98th bin, the 102nd bin, etc.). By performing the smoothing process in this way, frequency values concentrated in a certain bin are distributed to bins located in the vicinity of the bin. This makes it possible to reduce the influence of a slight deviation of the bins in which the frequency values are voted on the determination result.

The comparison unit 109 compares the leveled reference histogram (first leveled histogram), which is the leveled reference histogram, with the leveled detection histogram (second leveled histogram), which is the leveled detection histogram. and generates comparison information indicating the comparison result. A method for comparing both normalized histograms should not be particularly limited, and a known or novel method can be used as appropriate. The form of the comparison information is determined according to the comparison method employed, and may be, for example, the matching rate of both normalized histograms, the detection rate indicating the probability of occurrence of an abnormality (e.g., the reciprocal of the matching rate), and the like. .

The determination unit 110 performs right/wrong determination processing for determining whether or not the reference image and the detected image match based on the comparison result of the comparison unit 109 . The right/wrong determination process according to the present embodiment is performed by comparing comparison information with a threshold value. For example, if the comparison information is the detection rate (the higher the value, the higher the possibility of disagreement (abnormality)), if the detection rate is greater than a certain threshold value (for example, 20%), the reference image and the detection image are the same. It is determined that they do not match (specimen is abnormal). If the comparison information is a match rate (the smaller the value, the higher the possibility of mismatch), if the match rate is smaller than a certain threshold (e.g., 80%), it is determined that the reference image and the detected image do not match. be done.

The threshold value setting unit 111 sets a threshold value for correctness determination processing. Although the threshold setting method should not be particularly limited, it is preferable that the threshold can be arbitrarily changed by the user inputting a predetermined operation to the input unit 103, for example. At this time, it is preferable that the user can set the threshold while referring to the reference image, detection image, histogram, comparison information (detection rate, etc.) displayed on the display device 115 such as the display 15 . This allows the user to appropriately set the threshold. For example, it can be set so that the increase in detection rate (decrease in matching rate) due to changes in ambient light can be ignored, or the difference in sample placement (position within the detection area, sample orientation, etc.) can be ignored. It becomes possible to

The display control unit 112 performs processing for causing the display device 115 to display images (including still images and moving images) necessary for realizing the functions of the functional units 101 to 111 described above. The display control unit 112 according to the present embodiment includes various setting screens (detection area, reference image, detection image, screen for setting thresholds, etc.), current or past images captured by the imaging devices 2a to 2f, reference An image, a detected image, various histograms (reference histogram, detection histogram, leveled reference histogram, leveled detection histogram, etc.), comparison information (detection rate, match rate, etc.), determination results, etc. are displayed on the display device 115 .

The external output unit 113 outputs the information generated by the functional units 101 to 111 to the external device 5 . The external output unit 113 may output, for example, the comparison information generated by the comparison unit 109 and the determination result generated by the determination unit 110 to the external device 5 that constitutes the production facility or the like.

Note that the determination unit 110 and the threshold setting unit 111 do not necessarily have to be provided in the information processing device 3 , and may be realized using other information processing devices existing in the image processing system 1 . In this example, the reference image and the detection image are set by the reference image setting unit 105 and the detection image setting unit 106 from the images (video data) captured by the imaging devices 2a to 2f. The setting method of the detection image is not limited to this. The reference image and the detection image should be set by an appropriate method according to the hardware configuration of the image processing system 1, the properties of the specimen to be monitored, and the like.

Each of the above functional units 101 to 113 is realized by one or more integrated circuits, for example. Each of the functional units 101 to 113 may be implemented by causing a processor such as the CPU 11 to execute a program, that is, by software. Further, each of the functional units 101 to 113 may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Further, each of the functional units 101 to 113 may be implemented using both software and hardware. When multiple processors are used, each processor may implement one of the functional units 101-113, or may implement two or more of the functional units 101-113.

FIG. 4 is a flowchart showing an example of processing by the information processing device 3 according to the first embodiment. Steps S101 to S108 are setting processes performed before execution of the correct/incorrect judgment process. First, the region setting unit 104 performs processing for setting a detection region (S101).

FIG. 5 is a diagram showing an example of the state of the setting screen 151 according to the first embodiment when the area is set. The setting screen 151 of this example displays an image 152 shot by the second imaging device 2b (CAM2), and the image 152 is switched to an image shot by the first imaging device 2a (CAM1). It is made possible. The setting screen 151 is also provided with an indicator 153 for designating the time when the image 152 was shot. When the user operates the indicator 153 to designate a desired time, the image 152 corresponding to the time is displayed. A detection region 155 and three specimens 161a to 161c are displayed in the image 152 of this example. The detection area 155 is arbitrarily set by a user's operation. The detection region 155 in this example is set to include images of three specimens 161a-161c.

Thereafter, as shown in FIG. 4, the reference image setting unit 105 performs processing for setting a reference image (S102).

FIG. 6 is a diagram showing an example of the state of the setting screen 151 according to the first embodiment when the reference image is set. The setting screen 151 of this example includes a reference image display portion 171 , a detected image display portion 172 , a histogram display portion 173 , and a threshold value operation portion 174 . An image within the detection area 155 is displayed on the reference image display portion 171 of this example. The image within sensing region 155 is an image cropped from past or present video 152 . When the user presses an OK button 179 on the setting screen 151 , the image displayed on the reference image display portion 171 is stored in the storage portion 102 as the reference image 175 .

Thereafter, as shown in FIG. 4, the detection image setting unit 106 performs processing for setting a detection image (S103).

FIG. 7 is a diagram showing an example of the state of the setting screen 151 according to the first embodiment when the detection image is set. The image within the detection area 155 shown in FIG. 7 is an image cropped from the past or present video 152, similar to the detection area 155 shown in FIG. The image within the detection region 155 shown in FIG. 7 is different from the image within the detection region 155 shown in FIG. That is, the time when the video 152 shown in FIG. 7 was shot is different from the time when the video 152 shown in FIG. 6 was shot. An image within the detection area 155 shown in FIG. 7 is displayed in the detection image display section 172 of this example. When the user presses an OK button 179 on the setting screen 151 , the image displayed on the detected image display portion 172 is stored in the storage portion 102 as the detected image 176 .

Thereafter, as shown in FIG. 4, the histogram generation unit 107 generates a reference histogram, which is a histogram of pixel values in the reference image 175, and a detection histogram, which is a histogram of pixel values in the detection image 176 (S104). The leveling unit 108 generates a leveled reference histogram obtained by leveling the reference histogram and a leveled detection histogram obtained by leveling the detection histogram (S105).

FIG. 8 is a flow chart showing a specific example of processing by the histogram generating unit 107 and the leveling unit 108 according to the first embodiment. Here, an example of histogramming the values of the H (hue) component of HSV is shown.

The histogram generation unit 107 generates a reference histogram from the H component values of each pixel of the reference image 175, and generates a detection histogram from the H component values of each pixel of the detection image 176 (S151). Thereafter, the smoothing unit 108 multiplies the histogram vector H obtained by vectorizing both histograms generated by the histogram generating unit 107 and the smoothing matrix S to calculate a smoothed vector H' (S152). That is, the histogram vector H corresponding to the reference histogram is multiplied by the equalization matrix S to generate the equalization vector H′ corresponding to the reference histogram, and the histogram vector H and the equalization matrix S corresponding to the detection histogram are generated. A smoothed vector H' corresponding to the detected histogram is generated by the integration. The following formula (1) is an example of the histogram vector H, the following formula (2) is an example of the smoothing matrix S, and the following formula (3) is an example of the smoothing vector H'.

The examples of the above formulas (1) to (3) are examples in which the number of bins of the histogram is 5, and the sizes of the histogram vector H, the smoothing vector H', and the smoothing matrix S depend on the number of bins. Change. For example, when the number of bins is 256, the size of the histogram vector H and the smoothing vector H' is 256, and the size of the smoothing matrix S is 256×256.

After that, the smoothing unit 108 generates a smoothed histogram based on the smoothing vector H' (S153). That is, a leveled reference histogram is generated based on the leveled vector H' corresponding to the reference histogram, and a leveled detection histogram is generated based on the leveled vector H' corresponding to the detection histogram.

FIG. 9 is a diagram showing an example of the histogram 181 before smoothing according to the first embodiment. FIG. 10 is a diagram showing an example of a histogram 182 after smoothing according to the first embodiment. A histogram 181 (reference histogram or detection histogram) before smoothing corresponds to the histogram vector H in equation (1). A histogram 182 after smoothing (a smoothed reference histogram or a smoothed detection histogram) corresponds to the smoothed vector H' of equation (3).

In the histogram 181 before smoothing shown in FIG. It's becoming In the smoothed histogram 182 shown in FIG. 10, the first bin has a frequency value of 2/3, the second bin has a frequency value of 5/3, the third bin has a frequency value of 5/3, and the fourth bin has a frequency value of 5/3. has a frequency value of 1. That is, by the smoothing process, the frequency value 2 of the second bin and the frequency value 3 of the third bin before smoothing are distributed to the first bin and the fourth bin located near these bins.

Note that the method of distributing the frequency values by the smoothing process is not limited to the above, and should be appropriately adjusted according to the number of bins of the histogram, the characteristics of the target pixel value, and the like. The distribution of frequency values can be adjusted by changing the structure of the smoothing matrix S. FIG. By changing the configuration of the smoothing matrix S, it is possible to change, for example, the width of distribution of frequency values to other bins. The smoothing matrix S may be stored in advance in the storage unit 102, may be set by the user, or may be automatically and dynamically generated by a program, for example.

After that, as shown in FIG. 4, the comparison unit 109 compares both the leveled histograms (the leveled reference histogram and the leveled detection histogram) to calculate the detection rate (S106).

The method of comparing the leveled reference histogram and the leveled detection histogram (method of calculating the detection rate, match rate, etc.) should not be particularly limited, and known or novel methods can be used as appropriate. For example, calculation methods using correlation, chi-square, intersection, Bhattacharyya distance, etc. are known (reference: http://opencv.jp/opencv-2svn/cpp/histograms.html). The following formulas (4) and (5) are calculation formulas based on correlation, the following formula (6) is a calculation formula based on chi-square, the following formula (7) is a calculation formula based on intersection, and the following formula (8) is Bhattacharyya distance calculation formula. In the following equations, H1 represents one histogram (eg, the leveled reference histogram 201), H2 represents the other histogram (eg, the leveled detection histogram 202), and N represents the number of bins.

After that, the display control unit 112 displays the reference image 175, the detection image 176, the detection rate, the correct/incorrect judgment result, the histogram before and after the leveling, etc. on the display device 115 (S107). The threshold setting unit 111 performs processing for setting a threshold of the detection rate that serves as a criterion for correct/incorrect judgment according to the user's operation on the input unit 103 (S108). At this time, if the reference image 175 or the detection image 176 is changed, the displayed detection rate, correct/incorrect judgment result, histogram before and after leveling, etc. also change. As a result, the user can appropriately set the threshold of the detection rate while referring to the reference image 175, the detection image 176, the detection rate, the correctness determination result, the histogram before and after the leveling, etc. displayed on the display device 115. It becomes possible. The same applies to the thresholds of other comparison information such as matching rates.

FIG. 11 is a diagram showing an example of the first state during threshold setting on the setting screen 151 according to the first embodiment. In this example, both the reference image 175 and the detection image 176 include three specimens 161a to 161c. The arrangement of 161a to 161c is different.

The histogram display portion 173 displays a leveled reference histogram 201 corresponding to the reference image 175 and a leveled detection histogram 202 corresponding to the detection image 176 . Both normalized histograms 201 and 202 include peaks 201a and 202a corresponding to the hue of the first sample 161a, peaks 201b and 202b corresponding to the hue of the second sample 161b, and peaks 201b and 202b corresponding to the hue of the third sample 161c. It includes peaks 201c and 202c that There is some difference between both smoothed histograms 201, 202 for each peak 201a, 201b, 201b, 202b, 201c, 202c, which is due to the placement of each analyte 161a-161c. not due to absence.

As described above, in this embodiment, instead of comparing the reference histogram and the detection histogram, by comparing the leveled reference histogram 201 and the leveled detection histogram 202, the specimens 161a to 161c as described above are detected. It is possible to reduce the influence due to the difference in arrangement of . In addition, it is possible to reduce the influence of subtle changes in hue due to changes in ambient light. This makes it possible to reduce the influence of factors other than the abnormalities of the specimens 161a to 161c and appropriately perform the right/wrong determination.

In the example shown in FIG. 11, the threshold value of the detection rate is set to 2% as indicated by the threshold value operation unit 174 (when the detection rate is 2% or more, it is determined as abnormal (changed)). ). This threshold can be arbitrarily changed by the user operating the threshold operating unit 174 . The setting screen 151 includes a detection rate display portion 191 that indicates the result of the correctness determination. Since the detection rate in this example is 3%, which exceeds the threshold of 2%, it is determined here as "changed".

FIG. 12 is a diagram showing an example of the second state when setting the threshold on the setting screen 151 according to the first embodiment. In this example, the reference image 175 includes three specimens 161a to 161c, and the detection image 176 includes two specimens 161a and 161c. That is, the detected image 176 of this example does not include the specimen 161b. Accordingly, the leveled detection histogram 202 of this example includes a peak 202a corresponding to the sample 161a and a peak 202c corresponding to the sample 161c, but does not include a peak 202b corresponding to the sample 161b. Therefore, the detection rate in this example is a relatively high value of 40%, and the right/wrong determination is "changed".

The detection image 176 of the first example shown in FIG. 11 includes all three specimens 161a to 161c like the reference image 175, but the detection image 176 of the second example shown in FIG. Only two specimens 161a, 161c are included. In such a case, it may be appropriate to determine the detected image 176 of the first example as normal (no change) and the detected image 176 of the second example as abnormal (changed). Therefore, by operating the threshold operation unit 174 to set the threshold of the detection rate within the range of 3% to 40%, a state like the detection image 176 of the first example shown in FIG. 161c) is determined to be normal (no change), and a state like the detection image 176 of the second example shown in FIG. state) can be determined to be abnormal (changed).

13A and 13B are diagrams illustrating examples of correct/incorrect determination results according to the first embodiment. By setting the detection rate threshold to 3% to 40% as described above, the determination result for the detection image 176 of the first example on the upper right side is "no change", and the detection image 176 of the second example on the lower right side The determination result for the detected image 176 is "changed". Note that the method of setting the threshold is not limited to the above, and should be appropriately selected according to the properties of the specimen, the required determination accuracy, and the like.

FIG. 14 is a diagram showing an example of a third state during threshold setting on the setting screen 151 according to the first embodiment. The setting screen 151 according to this example includes a detection rate chart display portion 177 . The detection rate chart display portion 177 displays a chart showing changes in the detection rate of an image (detection image 176) within the detection area 155 set in the video 152 that changes in real time. In this example, the threshold for the detection rate is set to 20%, and when the detection rate, which changes in real time, becomes greater than the threshold (when there is no change to change) and when the detection rate becomes smaller than the threshold (at the point of transition from change to no change) is shown. The user may set the detection rate threshold while referring to such a chart and the image within the detection area 155 .

FIG. 15 is a diagram showing an example of the state of the setting screen 151 when setting is completed in the first embodiment. A setting content display portion 166 is included in the setting screen 151 when setting is completed. The setting content display section 166 displays information indicating the content of the setting performed as described above, and in this example, information such as the color components of the reference image 175 and the detection image 176 and the threshold value of the detection rate is displayed. . The color component in this example is "hue", which means that the images to be compared (the reference image 175 and the detection image 176) are color images, and the hue of each pixel constituting the color image is compared. It shows that the correctness judgment is performed.

As for the color component, in addition to "hue", "gray scale", "black and white binary", etc. may be selectable. When "grayscale" is selected, the image to be compared is a grayscale image, and the leveling reference histogram (reference histogram) and leveling detection histogram (detection histogram) are pixel values indicating grayscale gradation. (brightness value, density value, etc.). As a result, even when grayscale images are compared, processing can be performed in the same manner as described above.

After the above-described setting process (setting of the detection region 155, the reference image 175, the detection rate threshold, etc.) is completed, the determination unit 110 starts the actual right/wrong determination process (S109). It should be noted that the setting process may be performed during the execution of the actual right/wrong determination process.

According to the present embodiment, by comparing the leveled reference histogram 201 and the leveled detection histogram 202, it is possible to reduce the influence of factors other than the abnormalities of the specimens 161a to 161c and appropriately perform correct/incorrect determination. Become. In addition, since the threshold of the detection rate can be set while checking the reference image 175, the detection image 176, the smoothed histograms 201 and 202, the correct/incorrect judgment result, etc., the threshold can be set so that an appropriate correct/incorrect judgment is performed. can be set. This makes it possible to improve the practicality of correct/incorrect judgment.

Other embodiments will be described below with reference to the drawings, but portions that have the same or similar effects as those of the first embodiment may be denoted by the same reference numerals, and descriptions thereof may be omitted.

(Second embodiment)
The image processing system 1 (information processing device 3) according to the second embodiment includes a configuration suitable for the case where grayscale images are to be compared. FIG. 16 is a diagram showing an example of the first state during threshold setting on the setting screen 221 according to the second embodiment. FIG. 17 is a diagram showing an example of the second state when setting the threshold on the setting screen 221 according to the second embodiment.

The setting screen 221 according to this embodiment includes a difference image display section 225 . The difference image display portion 225 displays a difference image 226 (second image) in which the reference image 175 and the detection image 176 are superimposed.

The histogram generation unit 107 according to this embodiment generates a reference histogram that is the histogram of the reference image 175 and a difference histogram that is the histogram of the difference image 226 . At this time, the pixel values used to generate both histograms are lightness values or the like that indicate grayscale gradations. The leveling unit 108 according to the present embodiment generates a leveled reference histogram obtained by leveling the reference histogram and a leveled difference histogram obtained by leveling the difference histogram. The comparison unit 109 according to this embodiment calculates the detection rate by comparing the leveled reference histogram and the leveled difference histogram.

As shown in FIG. 16, the histogram display section 173 according to this embodiment displays a leveled reference histogram 231 and a leveled difference histogram 232 . The detection rate display portion 191 according to the present embodiment displays the detection rate indicating the comparison result between the leveled reference histogram 231 and the leveled difference histogram 232 .

As described above, in this embodiment, when the image to be compared is a grayscale image, the reference image 175 (leveled reference histogram) and the detection image 176 (leveled detection histogram) are compared. Instead, the reference image 175 (the normalized reference histogram) and the difference image 226 (the normalized difference histogram) are compared. As a result, even when dealing with a grayscale image that is difficult to differentiate by hue, it is possible to accurately determine whether the image is correct or not.

(Third embodiment)
FIG. 18 is a block diagram showing a functional configuration example of an information processing device 251 according to the third embodiment. An information processing device 251 according to this embodiment includes a recalculation unit 261 (exclusion unit) and an origin moving unit 262 in addition to the functional units 101 to 113 of the information processing device 3 according to the first embodiment.

The recalculator 261 recalculates the pixel values of the reference image 175 and the detection image 176 so as to exclude pixel values having a predetermined value from histogram targets. For example, when the pixel value is HSV, the S component value is equal to or less than a predetermined lower limit (saturation is equal to or less than the lower limit), or the V component value is equal to or less than a predetermined lower limit (lightness is equal to or less than the lower limit). value) or the value of the V component is greater than or equal to a predetermined upper limit value (the brightness is greater than or equal to the upper limit value) are invalidated.

The origin moving unit 262 moves the origins of the leveled reference histogram and the leveled detection histogram so that the peaks in the leveled reference histogram and the leveled detection histogram are not positioned within the end regions set in each histogram. . The edge area is an area consisting of an area near the origin and an area near the end point on the horizontal axis of the histogram. The method of setting the end region should not be particularly limited, and should be appropriately selected according to the usage situation. For example, in a 256-bin histogram, the 1st to 10th bin areas and the 247th to 256th bin areas may be the edge areas.

For example, when using pixel values that indicate a color space as the object of a histogram, if the peak is located in the edge area, the elements for comparison will be concentrated near the origin and near the end point, causing a problem in correctness judgment. may occur. Therefore, when the peak is located in the edge area, the origin of the histogram is moved so that the peak is located near the center of the histogram (for example, the 128th bin).

FIG. 19 is a flow chart showing an example of processing by the information processing device 251 according to the third embodiment. The difference between the flowchart according to the present embodiment shown in FIG. 19 and the flowchart according to the first embodiment shown in FIG. H.262 includes step S202.

After the setting of the reference image 175 and the detection image 176 is completed in steps S101 to S103, the recalculation unit 261 sets the reference image 175 and the detection image 176 so as to exclude pixel values having a predetermined value from histogram targets. The H component is recalculated (S201). After that, a reference histogram and a detection histogram are generated using the recalculated pixel values (H component) (S104), and a leveled reference histogram and a leveled detection histogram are generated from the reference histogram and the detection histogram (S105). .

After that, the origin moving unit 262 adjusts the origin position of each of the leveled reference histogram and the leveled detection histogram generated from the pixel values recalculated as described above so that the peak is not located within the end region. (S202). The comparison unit 109 compares the leveled reference histogram and the leveled detection histogram generated as described above to calculate the detection rate (S106).

FIG. 20 is a flowchart showing a specific example of processing by the recalculation unit 261 according to the third embodiment. When the recalculation unit 261 acquires the HSV component for each pixel of the reference image and the detection image (S251), it determines whether there is an unprocessed pixel (recalculation processing has not yet been performed) (S252). ).

If there is an unprocessed pixel (S252: Yes), the recalculation unit 261 determines whether S0≦S and V0≦V≦V1 holds for the HSV component of a certain pixel (S253). S is the value of the S component, S0 is the lower limit of the S component, V is the value of the V component, V0 is the lower limit of the V component, and V1 is the upper limit of the V component.

If S0≦S and V0≦V≦V1 are satisfied (S253: Yes), the recalculation unit 261 sets the pixel as a valid pixel (S254), and executes step S252 again. At this time, the recalculation unit 261 does not have to change the pixel values of valid pixels. On the other hand, if S0≤S and V0≤V≤V1 are not satisfied (S253: No), the recalculator 261 sets the pixel as an invalid pixel (S255), and executes step S252 again. When the above process is repeated and there are no more unprocessed pixels (S252: No), the recalculation unit 261 reflects invalid pixels on the images (video 152, reference image 175, detection image 176, etc.) (S256).

FIG. 21 is a diagram showing an example of the lower limit value S0 of the S component according to the third embodiment. FIG. 22 is a diagram showing examples of the lower limit value V0 and the upper limit value V1 of the V component according to the third embodiment. 21 and 22 illustrate a color space table 271 indicating the HSV color space. When the saturation (S) becomes lower than the lower limit value S0, the entire image becomes substantially black. Further, when the brightness (V) is lower than the lower limit value V0, the entire screen is substantially black, and when the brightness (V) is higher than the upper limit value V1, the entire screen is substantially white. Therefore, pixels for which S0≦S and V0≦V≦V1 do not hold can be invalidated.

FIG. 23 is a diagram showing an example of a state in which pixels are disabled in the third embodiment. In this example, S0=10, V0=30, and V1=100. That is, the effective range of saturation (S) is 10 or more, and the effective range of lightness is 30 or more and 100 or less. In the example shown in FIG. 23, among 3×3 pixels, pixels having S or V components outside the valid range are invalid. A histogram is generated using only the H component of valid pixels (those not crossed out in the right figure).

FIG. 24 is a diagram showing an example of the state of the setting screen 281 according to the third embodiment when the reference image is set. The setting screen 281 according to this example includes a post-change image display portion 285 . The post-change image display portion 285 displays the post-change image 286 in which the result of the invalidation process is reflected on the reference image 175 . In the post-change image 286, pixels in the background portion other than the specimens 161a to 161c are invalidated and turned black. When the user presses the OK button 179, the post-change image 286 displayed in the post-change image display section 285 is stored as the reference image. Although the reference image has been described here, the same applies to the detection image.

FIG. 25 is a diagram showing an example of the state of the setting screen 281 during determination processing according to the third embodiment. A reference image display portion 171 of the setting screen 281 according to the present example displays a reference image 288 changed by the invalidation process, and a detected image display portion 172 displays a detected image 289 changed by the invalidation process. It is The detected image 289 in this example does not include the specimens 161a-161c. The background portions of these reference image 288 and detection image 289 other than the specimens 161a to 161c are blackened by the invalidation process.

FIG. 26 is a flowchart showing a specific example of processing by the origin moving unit 262 according to the third embodiment. When the leveling unit 108 generates the leveled histogram h(i) (S281), the origin moving unit 262 sets i=0 (S282). i represents the histogram bin, eg i=0-255. h(i) represents the frequency value corresponding to each bin. After that, the origin moving unit 262 determines whether or not i≦256 is established (S283).

If i≦256 holds (S283: Yes), the origin moving unit 262 determines whether h(i−1)<h(i) and h(i)≧h(i+1) hold ( S284). where h(-1)=h(255) and h(256)=h(0). If h(i−1)<h(i) and h(i)≧h(i+1) are established (S284: Yes), the origin moving unit 262 adds i to the vertex position candidates (S285). That is, among three adjacent bins, if the middle bin h(i) is greater than the immediately preceding bin h(i−1) and is equal to or greater than the immediately succeeding bin h(i+1), i corresponds to the peak position. determined to be a possible bin. After that, the origin moving unit 262 adds 1 to i (S286), and executes step S283 again. On the other hand, if h(i−1)<h(i) and h(i)≧h(i+1) are not satisfied (S284: No), step S286 is executed without executing step S285.

When the above process is repeated and i ≤ 256 is no longer true (S283: No), the origin moving unit 262 calculates the average value of one or more i that are candidates for the vertex position (S287). is located within the end region of the histogram (S288). If the vertex position is located within the end region (S288: Yes), the origin moving unit 262 moves the origin position so that the vertex position is located in the center of the histogram (S289). For example, by shifting the origin by 128 bins in a 256-bin histogram, the peak can be placed approximately in the center of the histogram. If the vertex position is not located within the edge region (S288: No), the origin moving unit 262 ends the routine without moving the origin position.

FIG. 27 is a diagram showing an example of a histogram (leveled reference histogram or leveled detection histogram) before execution of origin movement processing according to the third embodiment. FIG. 28 is a diagram showing an example of a histogram after execution of origin movement processing according to the third embodiment. As shown in FIG. 27, if the histogram peak 291 is located near the origin, the elements for comparison are concentrated near the origin and the end point. Therefore, as shown in FIG. 28, by moving the origin of the histogram, the peak 291 is positioned at the center of the histogram.

According to the present embodiment, pixels that do not contribute to correctness determination of the specimens 161a to 161c can be invalidated. As a result, it is possible to reduce the computational load, etc., without impairing the accuracy of the correct/incorrect judgment. In addition, it is possible to prevent the elements for comparison from concentrating at the ends of the histogram, so it is possible to improve the accuracy of correct/incorrect judgment.

(Fourth embodiment)
FIG. 29 is a block diagram showing a functional configuration example of an information processing apparatus 301 according to the fourth embodiment. An information processing apparatus 301 according to this embodiment includes a color setting section 311 in addition to the functional units 101 to 113, 261, and 262 of the information processing apparatus 251 according to the third embodiment shown in FIG.

The color setting unit 311 sets a target color to be compared. The comparison unit 109 compares the leveled reference histogram and the leveled detection histogram generated for the set target color. Therefore, the right/wrong determination is performed only for the set target color. Although the method of setting the target color should not be particularly limited, for example, the user can specify an arbitrary color on the image 152 displayed on the display device 115 such as the display 15 using the mouse 18 or the like. may be broken.

FIG. 30 is a flow chart showing an example of processing by the information processing apparatus 301 according to the fourth embodiment. The difference between the flowchart according to this embodiment shown in FIG. 30 and the flowchart according to the third embodiment shown in FIG. at the point.

After the setting of the reference image 175 and the detection image 176 is completed in steps S101 to S103, the color setting unit 311 performs processing for setting the target color (S301). The method of setting the target color is not particularly limited, but is performed by designating an arbitrary color from images (image 152, reference image 175, detection image 176, etc.) displayed on the setting screen. After that, the processing after step S201 is performed.

FIG. 31 is a flow chart showing a specific example of processing by the color setting unit 311 according to the fourth embodiment. The color setting unit 311 first acquires the HSV components of the reference image 175 and the detected image 176 (S351), and sets the reference hue component h1, the reference saturation component s1, and the reference brightness component v1 based on the set colors. (S352). The reference hue component h1, the reference saturation component s1, and the reference brightness component v1 correspond to the H, S, and V components of the target color, respectively. After that, the color setting unit 311 determines whether or not there is an unprocessed pixel (S353).

If there is an unprocessed pixel (S353: Yes), the color setting unit 311 determines that h1-a≤H≤h1+b, s1-c≤S≤s1+d, and v1-e≤V≤v1+f for a certain pixel. It is determined whether or not (S354). a to f are constants. That is, the H component value (H), the S component value (S), and the V component value (V) of the target pixel are the reference hue component h1, the reference saturation component s1, and the reference brightness component, respectively. It is determined whether or not it is within the error range of v1.

If h1-a≤H≤h1+b, s1-c≤S≤s1+d, and v1-e≤V≤v1+f are established (S354: Yes), the color setting unit 311 sets the pixel as a valid pixel ( S355), and step S353 is executed again. On the other hand, if h1−a≦H≦h1+b, s1−c≦S≦s1+d, and v1−e≦V≦v1+f are not established (S354: No), the color setting unit 311 designates the pixel as an invalid pixel. set (S356), and step S353 is executed again. When the above process is repeated and there are no more unprocessed pixels (S353: No), the color setting unit 311 reflects invalid pixels on the image (video 152, reference image 175, detection image 176, etc.) (S357).

FIG. 32 is a diagram showing an example of the state of the setting screen 321 according to the fourth embodiment during color setting processing. This example shows an example in which a color is directly specified from the image 152 displayed on the setting screen 321 . In this example, the user can designate a desired color by means of a pointer 325 operated by the mouse 18 or the like. The setting screen 321 of this example includes a setting color display section 327 . The color of the pixel designated by the pointer 325 is displayed in the set color display portion 327 . When the user presses the OK button 179, the color displayed in the set color display section 327 is stored in the storage section 102 as the target color.

FIG. 33 is a diagram showing an example of the state when the target color is reflected on the setting screen 321 according to the fourth embodiment. In this example, in the image 152 displayed on the setting screen 321, pixels having colors other than the colors set as described above are disabled. In this way, the reference image and the detection image may be set from the image 152 in which the target color is reflected, or the color may be specified from the reference image and the detection image.

According to this embodiment, it is possible to narrow down the pixels for which histograms and images are to be generated to only pixels having hues desired by the user. This makes it possible to reduce the processing load.

(Fifth embodiment)
In the fourth embodiment, the color setting unit 311 sets the target color based on one color specified by the user on the image 152. However, the color setting unit 311 according to this embodiment The color range of the target color is set based on one or more colors specified by the user on a predetermined video (video 152, reference image 175, detected image 176, etc.) displayed on the display device 115. FIG. In this embodiment, an example of using HSL values as a color array (pixel values) indicating colors designated by the user is shown, but the type of color array is not limited to this. HSV values may be used as well as morphology.

FIG. 34 is a diagram showing an example of a setting screen 331 according to the fifth embodiment. The setting screen 331 according to the present embodiment includes an image display section 332 that displays an image 152 captured by any of the imaging devices 2a to 2f, a reference image display section 171 that displays a reference image 175, and a color range display section 333. contains.

The color range display section 333 is a section that displays the color range of the target color. The color range is the width of the color array (pixel value: each value of hue H, saturation S, and luminance L in this embodiment) recognized as the target color. The color range display section 333 according to the present embodiment includes a minimum value display section 341, a maximum value display section 342, a median value display section 343, an allowable error lower limit value setting section 344, and an allowable error upper limit value setting section 345. . The minimum value display portion 341 is a portion that displays the minimum value among a plurality of color arrays (HSL values) corresponding to each of a plurality of designated points designated by the user on a predetermined image. The maximum value display portion 342 is a portion that displays the maximum value among the plurality of color arrays corresponding to each of the plurality of designated points designated by the user. The intermediate value display section 343 displays the intermediate (average) value (for example, the value after the decimal point is rounded off, rounded up, or This is the part that displays the rounded down value). In addition, since the hue H takes a ring value in 256 gradations (0 to 255), the maximum value "18" in the example shown in FIG. , (168+273)/2=220.5. The allowable error lower limit value setting unit 344 is a part for setting the lower limit value of the allowable error range, and is configured so that the user can input an arbitrary value. The allowable error upper limit value setting unit 345 is a part for setting the upper limit value of the error allowable range, and is configured so that the user can input an arbitrary value.

FIG. 35 is a diagram showing an example of a state in which one specified point is specified on the setting screen 331 according to the fifth embodiment. When the user designates an arbitrary position (in this example, part of the handle of scissors) on the image 152 (for example, designates a position with a mouse and clicks), a first marker M1 is drawn at the position, and a first marker M1 is drawn at the position. The HSL values (254, 170, 87 in this example) of the pixels corresponding to the marker M1 of are obtained. At this point, only one specified point has been specified, so the same HSL values (254, 170, 87) are displayed in the minimum value display portion 341, the maximum value display portion 342, and the intermediate value display portion 343, respectively. is displayed. The reference image display section 171 displays a reference image 175 composed of pixels having HSL values from the lower limit of the allowable error to the upper limit of the allowable error. In the example shown in FIG. 35, a reference image 175 composed of pixels having an H value of 234 to 274 (234 to 255, 0 to 19), an S value of 150 to 190, and an L value of 67 to 107 is Is displayed.

FIG. 36 is a diagram showing an example of a state in which two specified points are specified on the setting screen 331 according to the fifth embodiment. When the user designates a second position on the image 152, a second marker M2 is drawn at that position, and the HSL values (247, 138, and 103 in this example) of pixels corresponding to the second marker M2 are acquired. be done. At this time, the minimum value display portion 341 displays a smaller value (H: 247, S: 138, and L: 87 in this example) in the comparison of the HSL values of both markers M1 and M2, and the maximum value display portion 342 displays larger values (H: 254, S: 170, and L: 103 in this example), and the median value display section 343 displays median values between the respective minimum and maximum values (this example H: 251, S: 154, and L: 95) are displayed. In this example, the lower limit of the allowable error is set to −20, and the upper limit of the allowable error is set to +20. The range is up to the added value. In the state illustrated in FIG. 36, the H value is in the range of 227 to 274 (227 to 255, 0 to 19), the S value is in the range of 118 to 190, and the L value is in the range of 67 to 123. A reference image 175 composed of pixels is displayed.

FIG. 37 is a diagram showing an example of a state in which three designated points are designated on the setting screen 331 according to the fifth embodiment. When the user designates a third position on the image 152, a third marker M3 is drawn at that position, and the HSL values (253, 183, and 82 in this example) of pixels corresponding to the third marker M3 are obtained. be done. At this time, the smallest value (H: 247, S: 138, and L: 82 in this example) among the HSV values of all the markers M1 to M3 is displayed in the minimum value display section 341, and the maximum value display section 342 , the largest value (H: 254, S: 183, and L: 103 in this example) is displayed, and the intermediate value display section 343 displays the intermediate value between the minimum and maximum values (in this example, H: 251, S: 154, and L: 95) are displayed. The reference image display unit 171 displays a reference image 175 composed of pixels having an H value of 227 to 274 (227 to 255, 0 to 19), an S value of 118 to 203, and an L value of 62 to 123. is displayed.

In the present embodiment, the three markers M1 to M3 are all positioned on the scissors handle, and the handle is assumed to have a color recognized as red. An example in which a position having a color different in hue from red is set as a designated point will be described below.

FIG. 38 is a diagram showing an example of a state in which four designated points are designated on the setting screen 331 according to the fifth embodiment. Here, a case where a position recognized as blue (part of an eraser) is designated as the fourth designated point is illustrated. When the user designates a fourth position on the image 152, a fourth marker M4 is drawn at that position, and the HSL values (168, 105, and 72 in this example) of pixels corresponding to the fourth marker M4 are acquired. be done. At this time, the minimum value display portion 341 displays the smallest value (H: 168, S: 105, and L: 72 in this example) among the HSL values of all the markers M1 to M4, and the maximum value display portion 342 , the largest value (H: 254, S: 183, and L: 103 in this example) is displayed, and the intermediate value display section 343 displays the intermediate value between the minimum and maximum values (in this example, H: 211, S: 144, and L: 88) are displayed. The reference image display unit 171 displays a reference image 175 composed of pixels having an H value of 148 to 274 (148 to 255, 0 to 19), an S value of 85 to 203, and an L value of 52 to 123. is displayed. In this case, the reference image 175 includes a red area 305 recognized as red, a blue area 306 recognized as blue, and a different color area 307 recognized as neither red nor blue (for example, gray). to be included. This is because the color range of the target color is greatly expanded by designating blue, which has a hue that is significantly different from red.

As described above, if a plurality of colors (eg, red and blue, etc.) with relatively large hue differences (eg, the difference in gradation value is equal to or greater than a predetermined value) are specified at the same time, an unintended color (eg, gray, etc.) may be displayed. may be included in the color range of the target color. One way to deal with such problems is to remove inappropriate designated points.

FIG. 39 is a diagram showing an example of a state when deleting designated points on the setting screen 331 according to the fifth embodiment. FIG. 40 is a diagram showing an example of the state after the designated point is deleted on the setting screen 331 according to the fifth embodiment. Here, a case of deleting the designated point corresponding to the fourth marker M4 is illustrated. As shown in FIG. 39, when the user re-specifies the position of the fourth marker M4 on the image 152 by a click operation or the like, the fourth marker M4 is deleted from the image 152 as shown in FIG. The display in 333 and the reference image 175 return to the state (the state shown in FIG. 37) before specifying the position corresponding to the fourth marker M4. Thus, by allowing the user to arbitrarily delete the specified point, it is possible to appropriately and simply adjust the color range of the target color.

Further, in this embodiment, the user can adjust the lower limit of the allowable error and the upper limit of the allowable error to arbitrary values.

FIG. 41 is a diagram showing an example of a state when adjusting the allowable error lower limit value and the allowable error upper limit value on the setting screen 331 according to the fifth embodiment. In this example, all the lower allowable error limits corresponding to each HSL value are changed from -20 to -4, the upper allowable error limit corresponding to the H value is changed from +20 to +7, and the allowable error corresponding to the S value is The upper limit is changed from +20 to +8, and the allowable error upper limit corresponding to the L value is changed from +20 to +8. As a result, the color range of the target color is H value: 243 to 261 (243 to 255, 0 to 6), S value: 134 to 191, and L value: 78 to 111. A reference image 171 is displayed that includes a red region 305' composed of pixels having the adjusted HSL values. In this way, by enabling the user to arbitrarily adjust the lower limit of the allowable error and the upper limit of the allowable error, it becomes possible to adjust the color range of the target color more appropriately and easily.

FIG. 42 is a flow chart showing an example of processing by the information processing apparatus 301 according to the fifth embodiment. Here, a processing example in which the user designates a plurality of designated points on the image 152 and sets the color range of the target color based on the HSL value corresponding to each designated point as described above will be described. The color setting unit 311 (see FIG. 29) according to the present embodiment first reads initial values relating to the process of setting the color range of the target color from the storage unit 101 (S501). The initial value includes an upper limit (for example, 10) of designated points that the user can designate on the image 152 . After that, the color setting unit 311 receives a click operation for specifying an arbitrary position on the image 152 by the user via the user interface (S502).

The color setting unit 311 determines whether a marker exists at the position clicked by the user on the image 152 (S503). If no marker exists at the clicked position (S503: No), the color setting unit 311 determines whether the number of markers (the number of clicks or the number of designated points) has reached the upper limit (S504). If the number of markers has not reached the upper limit (S504: No), the color setting unit 311 adds a marker to the clicked position (S505), and sets the pixel color array (HSL value) is obtained (S506), and the color range is calculated as described above based on the obtained one or more color arrays (S510). On the other hand, if the number of markers has reached the upper limit (S504: Yes), for example, if 10 markers have already been placed on the image 152, no new marker is added and the existing markers are added. A color range is calculated based on each color array corresponding to (S510).

If there is a marker at the clicked position (S503: Yes), the color setting unit 311 deletes the marker at the clicked position (S507), and deletes the color array corresponding to the deleted marker (S508). ). After that, the color setting unit 311 determines whether or not there are one or more markers on the image 152 (S509). If there is one or more markers (S509: Yes), the color range is calculated based on the color array corresponding to each marker (S510).

After that, the color setting unit 311 updates the target color image (the reference image 175 in the examples shown in FIGS. 34 to 41) composed of pixels having a color array included in the calculated color range (S511), It is determined whether or not there is a request for color specification from the user (S512). If there is no request (S512: No), this routine is terminated, and if there is a request (S512: Yes), the process returns to step S502.

In the above, the problem that occurs when a plurality of colors (for example, red and blue) with relatively large differences in hue are used as designated points (problem that an unintended color is included in the color range of the target color) is An example of solving the problem by deleting the designated point corresponding to one color (for example, blue) has been shown, but the embodiment is not limited to this. For example, if one or more specified points are specified for a red portion and one or more specified points are specified for a blue portion, the color range may be divided into two areas, a red area and a blue area. Also, the color range may be divided into three or more regions.

That is, the color range may be divided into a plurality of areas and set according to the characteristics of the color array of the plurality of designated points. For example, when there are a plurality of designated points, the difference in gradation value between pixel values (e.g., H value) indicating hue is a predetermined value or more (e.g., when the difference in H value of 256 gradations is 50 or more, etc.) ) may be set by dividing the color range into a plurality of areas.

For example, as three specified points with HSV values as a color array, the first specified point (65, 34, 124), the second specified point (232, 230, 200), and the third specified point (45, 64, 98) are specified, the lower limit of the allowable error is -10, and the upper limit of the allowable error is +10. In this case, based on the first designated point (65, 34, 124) and the third designated point (45, 64, 98), the H value range is 35 to 75, and the S value range is 24 to 74. , a first color range in which the V value range is 88 to 134, and based on the second designated point (232, 230, 200), the H value range is 222 to 242, and the S value range is A second color range can be established that ranges from 220-240 and V values from 190-210. Note that the algorithm for dividing and setting the color range is not limited to the above, and should be appropriately determined according to usage conditions and the like.

As described above, according to this embodiment, the color range of the target color is set based on one or more colors specified by the user. This makes it possible to set the target color more easily.

(Sixth embodiment)
FIG. 43 is a block diagram showing a functional configuration example of an information processing device 351 according to the sixth embodiment. An information processing device 351 according to this embodiment includes an expansion unit 361 in addition to the functional units 101 to 113, 261, 262, and 311 of the information processing device 301 according to the fourth or fifth embodiment shown in FIG. ing.

The expansion unit 361 performs expansion processing for expanding peaks in the leveled reference histogram and the leveled detection histogram. The comparison unit 109 compares an extended reference histogram (first extended histogram), which is an extended leveled reference histogram, and an extended detection histogram (second extended histogram), which is an extended leveled detection histogram. ) to generate comparison information indicating the comparison result.

Although the method of expansion processing should not be particularly limited, by expanding or contracting the peak width in the horizontal axis (bin axis) direction or by expanding or contracting the peak height in the vertical axis (frequency value axis) direction, , the peak shape is preferably expanded in the horizontal direction so that no gap is formed in the peak.

FIG. 44 is a flow chart showing an example of processing by the information processing device 351 according to the sixth embodiment. The difference between the flowchart according to this embodiment shown in FIG. 44 and the flowchart according to the fourth embodiment shown in FIG. It is in the point of being.

After the origin position of the histogram is adjusted by the origin moving unit 262 in step S202, the expansion unit 361 expands the leveled histograms (leveled reference histogram and leveled detection histogram) (S401). After that, the comparison unit 109 compares both extended histograms (extended reference histogram and extended detection histogram) to calculate the detection rate (S402). After that, the processing after step S107 is performed.

FIG. 45 is a flowchart showing a specific example of processing by the expansion unit 361 according to the sixth embodiment. The expansion unit 361 acquires the smoothed vector H′ (generated in step S152 in FIG. 8) (S451), and generates an expanded matrix E corresponding to the peak in the histogram corresponding to the smoothed vector H′. (S452). After that, the expansion unit 361 multiplies the smoothing vector H' and the expansion matrix E to generate the expansion vector H'' (S453).

Equation (9) below is a first example of the smoothing vector H′, Equation (10) below is a first example of the extended matrix E, and Equation (11) below is a first example of the extended vector H″. This is the first example.

The examples of the above equations (9) to (11) have four bins in the histogram, and the peaks are in the first and second bins of the smoothed histogram corresponding to the smoothing vector H′. formed.

Equation (12) below is a second example of the smoothed vector H′, Equation (13) below is a second example of the extended matrix E, and Equation (14) below is a second example of the extended vector H″. This is the second example.

In the example of equations (12)-(14) above, the number of bins in the histogram is 4 and the peak is formed in the second bin portion of the smoothed histogram corresponding to the smoothing vector H'.

The sizes of the smoothed vector H', the expanded vector H'', and the expanded matrix E change according to the number of bins. Further, the extended matrix E changes according to the peak position, peak width, peak height, etc. of the leveling vector H'. The augmented matrix E may be set by the user, or may be automatically and dynamically generated by the program.

After that, the expansion unit 361 generates an expanded histogram based on the expansion vector H'' (S454). That is, the extended reference histogram is generated based on the extended vector H'' corresponding to the leveled reference histogram, and the extended detection histogram is generated based on the leveled vector H'' corresponding to the leveled detection histogram.

FIG. 46 is a diagram illustrating an example of a histogram before expansion according to the fifth embodiment; FIG. 47 is a diagram showing an example of a histogram after expansion according to the fifth embodiment. The histogram before expansion shown in FIG. 46 corresponds to the leveled reference histogram or the leveled detection histogram. The extended histogram shown in FIG. 47 corresponds to the extended reference histogram or the extended detection histogram.

FIG. 46 illustrates a polygonal line 370 including three peaks 371a-371c before dilation and an expansion region 373 before expansion. The extended region 373 is a rectangular region set to include three peaks 371a-371c. The extended region 373 in this example is formed by line segments connecting point a (0, 0), point b (30, 0), point c (0, 28000), and point d (40, 28000). The three peaks 371a-371c in this example are concentrated in some bins (bins near the origin in this example) out of all the bins. Such a state occurs, for example, when the ratio of the area occupied by the specimens 161a to 161c (objects having hues) to the area of the entire detection region 155 is small. When comparison is performed using such a histogram, accuracy of right/wrong determination may be lowered.

FIG. 47 illustrates a polygonal line 370' including three peaks 371a'-371c' after dilation and an expanded region 373' after dilation. The expanded region 373' after expansion connects point a' (0, -450), point b' (250, -450), point c' (0, 3400), and point d' (250, 3400). It is formed by line segments. Thus, the expansion process in this example is performed without changing the total number of bins (256). Such dilation processing distributes the elements that make up the polygonal line 370', including the three peaks 371a'-371c', over the entire histogram. By distributing the elements to be compared, the accuracy of correct/incorrect judgment can be improved. The size, shape, and amount of expansion of the expansion regions 373 and 373' are merely examples, and the method of setting the expansion regions 373 and 373' should be appropriately selected according to the usage conditions.

FIG. 48 is a diagram showing an example of the state before expansion of the setting screen 371 according to the sixth embodiment. The reference image 175 in this example is an image showing only the background 374 . The detection image 176 of this example is an image in which one specimen 161 a exists on the background 374 . In the detection image 176 of this example, the ratio of the area of the specimen 161a to the area of the entire detection image 176 is considerably small.

In the histogram display portion 173 before expansion, the histograms before expansion, that is, the leveled reference histogram 375 corresponding to the reference image 175 and the leveled detection histogram 376 corresponding to the detection image 176 are displayed. The smoothed reference histogram 375 includes a peak 380 a corresponding to the hue of the background 374 . The smoothed detection histogram 376 includes a peak 380b corresponding to the hue of the background 374 and a peak 381 corresponding to the hue of the analyte 161a. The ratio of the elements that make up peak 381 to the total elements that make up smoothed detection histogram 376 is rather small. Therefore, the difference between the leveled reference histogram 375 and the leveled detection histogram 376 is slight, and the detection rate value displayed in the detection rate display section 191 is a considerably low value of 4.0%.

FIG. 49 is a diagram showing an example of the state after expansion of the setting screen 371 according to the sixth embodiment. The extended histogram display section 173 displays the extended histogram, that is, an extended reference histogram 391 obtained by extending the leveled reference histogram 375 and an extended detection histogram 392 obtained by extending the leveled detection histogram 376. is displayed. Dilated reference histogram 391 includes a dilated peak 380 a ′ corresponding to the hue of background 374 . Dilated detection histogram 392 includes dilated peak 380b' corresponding to the hue of background 374 and dilated peak 381' corresponding to the hue of analyte 161a. As shown in FIG. 49, the difference between the extended reference histogram 391 and the extended detection histogram 392 is quite large, and the detection rate value displayed in the detection rate display section 191 is a considerably large value of 69.0%. there is

As described above, by performing the expansion process on the smoothed histogram, even if the difference between the reference image 175 and the detection image 176 is small, the histogram corresponding to the reference image 175 and the detection image 176 can be matched. It is possible to increase the difference from the histogram to be used. This makes it possible to improve the sensitivity of correct/incorrect judgment. The degree of expansion may be adjustable by the user. For example, when an operation for increasing the sensitivity is input to the input unit 103, the expansion unit 361 generates the expansion matrix E so that the degree of expansion is increased. As a result, the user can check the reference image 175, the detection image 176, the histograms 375, 376, 391, 392, the detection rate, etc. displayed on the setting screen 371, and appropriately set the threshold of the detection rate and the degree of expansion. Adjustment is possible. This makes it possible to further improve the practicality of correct/incorrect judgment.

In addition, in the description of the sixth embodiment, an example in which the recalculation unit 261, the origin moving unit 262, and the color setting unit 311 are included is shown. , and the color setting unit 311 may not be included.

In the first to sixth embodiments, the reference image and the detection image are set based on arbitrary areas of the image indicated by the image data, but the present invention is not limited to such embodiments. For example, a still image captured by a still image capturing device such as a digital camera may be used as the reference image and the detected image. may be set based on the video data, or vice versa.

When at least part of various functions of the image processing system 1 and the information processing apparatuses 3, 251, 301, and 351 according to the first to sixth embodiments are realized by executing a program, the program is pre-installed in a ROM or the like. and provided. Also, the program may be recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD in the form of an installable or executable file and provided. Alternatively, the program may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network. Also, the program may be configured to be provided or distributed via a network such as the Internet. Also, the program may have a module configuration including at least part of each function described above.

Although the embodiment of the present invention has been described above, the above embodiment is presented as an example and is not intended to limit the scope of the invention. The novel embodiments described above can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. The above-described embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

1 image processing system 2a to 2f imaging device 3, 251, 301, 351 information processing device 4 network 5 external device 11 CPU
12 ROMs
13 RAM
14 external storage device 15 display 16 network I/F
17 keyboard 18 mouse 19 DVD drive 20 DVD
21 External device I/F
22 speaker 101 video reception unit 102 storage unit 103 input unit 104 area setting unit 105 reference image setting unit 106 detection image setting unit 107 histogram generation unit 108 equalization unit 109 comparison unit 110 determination unit 111 threshold setting unit 112 display control unit 113 external Output unit 115 Display device 151, 221, 281, 321, 331, 371 Setting screen 152 Video 153 Indicator 155 Detection area 161a to 161c Specimen 166 Setting content display unit 171 Reference image display unit 172 Detection image display unit 173 Histogram display unit 174 Threshold Operation section 175, 288 Reference image 176, 289 Detection image 177 Detection rate chart display section 179 OK button 181 Histogram before leveling 182 Histogram after leveling 191 Detection rate display section 201, 231 Leveling reference histogram 202 Leveling detection histogram 201a, 201b, 201c, 202a, 202b, 202c, 291, 371a to 371c, 371a' to 371c', 380a, 380a', 380b, 380b', 381, 381' peak 225 difference image display unit 226 difference image 232 leveling Difference histogram 261 Recalculation unit 262 Origin moving unit 271 Color space table 285 Post-change image display unit 286 Post-change image 305, 305' Red region 306 Blue region 307 Other color region 311 Color setting unit 325 Pointer 327 Set color display unit 332 Video Display unit 333 Color range display unit 341 Minimum value display unit 342 Maximum value display unit 343 Intermediate value display unit 344 Allowable error lower limit value setting unit 345 Allowable error upper limit value setting unit 361 Expansion unit 370, 370' Broken lines 373, 373' Expansion Region 374 Background 391 Extended Reference Histogram 392 Extended Detection Histogram M1-M4 Markers

JP 2006-194869 A

Claims (19)

a histogram generator that generates a first histogram that is a histogram of pixel values of the first image and a second histogram that is a histogram of pixel values of the second image;
a leveling unit that performs a leveling process of distributing the frequency values in the first histogram and the second histogram to one or more bins located near the bin corresponding to the frequency values;
a comparison unit that generates comparison information indicating a comparison result between a first smoothed histogram that is the smoothed first histogram and a second smoothed histogram that is the smoothed second histogram; ,
An image processing device comprising: a determination unit that performs determination processing for determining whether the first image and the second image match based on the comparison information and the threshold;
a threshold setting unit that sets the threshold;
The image processing apparatus according to claim 1, further comprising: an exclusion unit that excludes the pixel value having a predetermined value from the target of the histogram;
The image processing apparatus according to claim 1 or 2, further comprising: the pixel value includes an HSV component;
The value of the histogram is the value of the H component,
The exclusion unit determines whether the value of the S component is equal to or less than a predetermined lower limit, the value of the V component is equal to or less than a predetermined lower limit, or the value of the H component is equal to or greater than a predetermined upper limit. to exclude from said subject,
The image processing apparatus according to claim 3. moving the origins of the first smoothed histogram and the second smoothed histogram so that the peaks in the first smoothed histogram and the second smoothed histogram are not located within a predetermined edge area; origin moving part,
The image processing apparatus according to any one of claims 1 to 4, further comprising: a color setting unit for setting a target color;
further comprising
The comparison unit compares the first smoothed histogram and the second smoothed histogram generated using the pixel values corresponding to the target color.
The image processing apparatus according to any one of claims 1 to 5. The color setting unit sets the target color based on a color specified by a user on a predetermined image displayed on a display device.
The image processing apparatus according to claim 6. The color setting unit sets the color range of the target color based on one or more colors specified by the user.
The image processing apparatus according to claim 7. The color setting unit divides the color range into two or more and sets them when a plurality of colors having a gradation value difference between pixel values equal to or greater than a predetermined value are specified.
The image processing apparatus according to claim 8. The smoothing unit multiplies the vector of the first histogram and a smoothing matrix for performing the smoothing process to generate the vector of the first smoothed histogram, and the vector of the second histogram. generating a vector of the second smoothed histogram by multiplying the vector with the smoothing matrix;
The image processing device according to any one of claims 1 to 9. An expansion unit that performs expansion processing for expanding peaks in the first smoothed histogram and the second smoothed histogram,
further comprising
The comparison unit compares a result of comparison between a first expanded histogram, which is the first expanded histogram, and a second expanded histogram, which is the second expanded histogram. generating said comparison information indicative of
The image processing device according to any one of claims 1 to 10. The expansion unit multiplies the vector of the first smoothed histogram and an expansion matrix for performing the expansion process to generate a vector of the first expanded histogram, and performs the second smoothing process. generating the second expanded histogram vector by multiplying the expanded histogram vector with the expanded matrix;
The image processing apparatus according to claim 11. The extended matrix is dynamically generated according to the position of the peak,
The image processing apparatus according to claim 12. If the images to be compared are grayscale,
The first image is a reference image that serves as a reference for comparison,
wherein the second image is a difference image in which the reference image and the detection image to be compared are superimposed;
The image processing device according to any one of claims 1 to 13. an acquisition unit that acquires video data;
an image setting unit that sets at least one of the first image and the second image based on the video data;
The image processing apparatus according to any one of claims 1 to 14, further comprising: An image processing device according to claim 15;
an imaging device that generates the video data;
with
the acquisition unit acquires the video data from the imaging device;
image processing system. including a plurality of the imaging devices;
The image setting unit sets the first image or the second image from current or past images captured by the plurality of imaging devices.
17. The image processing system according to claim 16. generating a first histogram of the pixel values of the first image and a second histogram of the pixel values of the second image;
performing a smoothing process of distributing the frequency values in the first histogram and the second histogram to one or more bins located near the bin corresponding to the frequency values;
generating comparison information indicating a comparison result between a first smoothed histogram that is the smoothed first histogram and a second smoothed histogram that is the smoothed second histogram;
An image processing method including to the computer,
a process of generating a first histogram, which is a histogram of pixel values of the first image, and a second histogram, which is a histogram of pixel values of the second image;
A smoothing process of distributing the frequency values in the first histogram and the second histogram to one or more bins located near the bin corresponding to the frequency values;
A process of generating comparison information indicating a comparison result between a first smoothed histogram that is the smoothed first histogram and a second smoothed histogram that is the smoothed second histogram;
program to run.

Images