JP7223194B1 - Information processing method, computer program, information processing device and information processing system - Google Patents

Abstract

The present invention provides an information processing method, a computer program, an information processing device, a photographing device, and an information processing system that can be expected to improve the accuracy of detecting the state of an object based on a photographed image. In the information processing method according to the present embodiment, an information processing device acquires a first image of an object based on light in a first wavelength band, and captures the object in a second wavelength band. Obtain a second image taken based on light, perform segmentation on the first image to identify the image area where the target object is captured, and based on the identification result, identify the target object from the second image. Detect conditions. The first wavelength band may be a visible light wavelength band, and the second wavelength band may be an infrared wavelength band. The information processing device may classify the object based on the first image, and detect the state of the object from the second image based on the classification result and the identification result of the object. . [Selection diagram] Figure 1

Description

The present invention relates to an information processing method, a computer program, an information processing apparatus, and an information processing system for detecting the state of an object.

2. Description of the Related Art Conventionally, there has been widely used a technique of detecting the presence or absence of an abnormality or the degree of overripe of an object such as food based on an image of the object captured by a photographing device. In recent years, the state of an object is detected from a photographed image using a learning model that has undergone machine learning in advance, so-called AI (Artificial Intelligence).

In Patent Literature 1, a plurality of captured images captured by an imaging device so that an object is captured in different positions or postures are used to determine the relative positions or postures of the imaging device and the target in each of the plurality of captured images. A relationship is acquired in a manner that can be specified, a plurality of abnormality candidate images in which abnormality candidates that may correspond to the abnormality of the surface of the object are captured are extracted from the plurality of captured images, and a plurality of abnormality candidate images are extracted. estimating the depth of each abnormality candidate in the plurality of candidate abnormality images with respect to the surface of the object based on the change in the abnormality candidate between the plurality of candidate images; An anomaly detection device has been proposed for detecting whether or not

JP 2022-48904 A

However, there is a problem that it is difficult to accurately detect the state of an object such as an avocado that has a dark color or an object that has a pattern on the surface such as an apple.

The present invention has been made in view of such circumstances, and its object is to provide an information processing method, a computer program, and an information processing method that can be expected to improve the accuracy of detecting the state of an object based on a photographed image. An object of the present invention is to provide a processing device and an information processing system.

In an information processing method according to an embodiment, an information processing device acquires a first image of an object captured using light in a first wavelength band, and captures the object based on light in a second wavelength band. segmentation is performed on the first image to identify an image region in which the object is captured, and based on the identification result, an image region of the object is extracted from the second image. , a plurality of learning models machine-learned to classify the object based on the first image and output information about the state of the object in response to an input of an image region extracted from the second image; One learning model is selected from among according to the classification result, and the state of the object is detected based on the extracted image region and the selected learning model .

According to one embodiment, it can be expected to improve the accuracy of object state detection based on a captured image.

1 is a schematic diagram for explaining an overview of an information processing system according to an embodiment; FIG. 1 is a block diagram showing the configuration of an information processing apparatus according to an embodiment; FIG. 1 is a schematic diagram showing the configuration of a camera according to an embodiment; FIG. It is an image showing an example of a visible light image and an infrared image of an object photographed by a camera. FIG. 4 is a schematic diagram for explaining segmentation processing performed by an information processing apparatus; FIG. 4 is a schematic diagram for explaining image region extraction processing performed by an information processing apparatus; FIG. 3 is a schematic diagram for explaining anomaly detection processing performed by an information processing device; FIG. 11 is a schematic diagram showing an example of a notification screen of an abnormal location; 4 is a flowchart showing the procedure of state detection processing performed by the information processing apparatus according to the embodiment; 9 is a flow chart showing the procedure of state detection processing performed by the information processing apparatus according to the second embodiment; 10 is a flow chart showing the procedure of state detection processing performed by the information processing apparatus according to the third embodiment; FIG. 13 is a flow chart showing the procedure of state detection processing performed by the information processing apparatus according to the fourth embodiment; FIG. FIG. 13 is a flow chart showing the procedure of state detection processing performed by the information processing apparatus according to the fourth embodiment; FIG.

A specific example of an information processing system according to an embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to these examples, but is indicated by the scope of the claims, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

<System Overview>
FIG. 1 is a schematic diagram for explaining an outline of an information processing system according to this embodiment. The information processing system according to the present embodiment is a system that inspects an object 100 such as food or medicine by performing a process of detecting an abnormality or defect in the object 100 . The information processing system according to the present embodiment includes an information processing device 1 that performs processing such as control and calculation related to abnormality detection, a camera 3 that captures an image of an object 100, and a display device that displays information such as the results of abnormality detection. 5.

In the system configuration example shown in FIG. 1, the information processing system according to the present embodiment includes an inspection table 7 on which an object 100 for abnormality detection is placed. The camera 3 is held, for example, by an arm extending upward from the table of the inspection table 7, and can photograph the object 100 placed on the inspection table 7 from above. The information processing device 1 is arranged, for example, in the vicinity of the examination table 7, and is connected to the camera 3 and the display device 5 via communication lines, signal lines, or the like. The display device 5 is arranged, for example, on the inspection table 7, and presents information on inspection results such as presence/absence of abnormalities in the object 100 to a user who performs inspection work such as replacement of the object 100. .

Note that the information processing system according to the present embodiment is not limited to the configuration shown in FIG. The information processing system is not for the object 100 placed on the inspection table 7, but for example, the object 100 carried by a belt conveyor, the object 100 held by the robot arm, or the object 100 held by the user. The configuration may be such that the camera 3 photographs the target object 100 or the like. The camera 3 may be configured to photograph the object 100 not from above but from the side or below. Also, the camera 3 may be of a portable type instead of being fixed to the inspection table 7 . The camera 3 may exchange information with the information processing apparatus 1 by wireless communication instead of performing wired communication, and the camera 3 and the information processing apparatus 1 may be integrated.

The camera 3 included in the information processing system according to the present embodiment can switch between two wavelength bands such as a visible light region and an infrared region, for example. The camera 3 captures images in the visible light wavelength band by capturing images through an optical filter that transmits wavelengths in the visible light range but does not transmit wavelengths in the infrared range, for example. Hereinafter, an image obtained by the camera 3 capturing an image in the visible light region in this manner is referred to as a visible light image. In addition, the camera 3 performs photographing in the infrared wavelength band by photographing through an optical filter that transmits wavelengths in the infrared region but does not transmit wavelengths in the visible light region, for example. Hereinafter, an image obtained by the camera 3 capturing an image in the infrared region is called an infrared image. The camera 3 can switch between photographing in the visible light region and photographing in the infrared region by appropriately switching among these plural types of optical filters. The camera 3 captures an image of one object 100 with wavelengths in the visible light range and with wavelengths in the infrared range, and provides two images, a visible light image and an infrared image captured at each wavelength, as information. It is given to the processing device 1 .

The information processing apparatus 1 according to the present embodiment acquires two types of images, a visible light image and an infrared image, of the target object 100 photographed by the camera 3, and determines the image of the target object 100 based on the acquired two types of images. Perform processing to detect anomalies. In the present embodiment, the information processing apparatus 1 uses a visible light image of the object 100 to identify an image region (pixels) in which the object 100 is shown in the image, a so-called segmentation of the object 100. process. The information processing device 1 may also perform a process of determining what the target object 100 is using the visible light image, that is, a so-called class classification process of the target object 100 . Note that the information processing apparatus 1 according to the present embodiment performs these segmentation processing, class classification processing, and the like using a learning model that has undergone machine learning in advance, that is, so-called AI.

The information processing apparatus 1 can specify the image area in which the object 100 is shown in the infrared image by specifying the image area in which the object 100 is shown in the visible light image by the segmentation process. . In the present embodiment, one camera 3 is configured to photograph the (immobile) object 100 placed on the inspection table 7 by switching the optical filters. The photographed position is the same as the position at which the object 100 is photographed in the infrared image. If the target object 100 is moving by a belt conveyor or the like, the information processing device 1 determines based on the moving speed of the target object 100 by the belt conveyor and the time between capturing the visible light image and capturing the infrared image. , the amount of displacement between the position of the object 100 in the visible light image and the position of the object 100 in the infrared image can be estimated.

The information processing apparatus 1 performs processing for extracting the specified image area from the infrared image based on the result of specifying the image area in which the target object 100 is captured. The information processing device 1 detects abnormality of the object 100 based on the extracted image area. The information processing apparatus 1 according to the present embodiment performs abnormality detection of the target object 100 based on an image area extracted from an infrared image using a learning model, so-called AI, which has undergone machine learning in advance. The learning model receives, as an input, an image area of the object 100 extracted from an infrared image, for example, and numerical information indicating whether or not the object 100 captured in this image area has an abnormality, or Machine learning is performed in advance so as to output coordinate information or the like indicating 100 abnormal locations.

The information processing device 1 acquires information such as the presence or absence of an abnormality in the object 100 output by the learning model, and displays the acquired information on the display device 5 as an inspection result. Note that the information processing device 1 may display information on the display device 5 only when a result is obtained that the object 100 has an abnormality. In addition, in the information processing system according to the present embodiment, the information processing device 1 and the display device 5 are separate devices, but the present invention is not limited to this. good too.

<Device configuration>
FIG. 2 is a block diagram showing the configuration of the information processing device 1 according to this embodiment. The information processing apparatus 1 according to the present embodiment includes a processing unit 11, a storage unit (storage) 12, a communication unit (transceiver) 13, an operation unit 14, and the like. In this embodiment, one information processing apparatus 1 performs the processing, but a plurality of information processing apparatuses may perform the processing in a distributed manner.

The processing unit 11 includes an arithmetic processing unit such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit) or a quantum processor, a ROM (Read Only Memory) and a RAM (Random Access Memory). It is configured using The processing unit 11 reads and executes a program 12 a stored in the storage unit 12 to perform various processes related to detection of anomalies in the target object 100 .

The storage unit 12 is configured using, for example, a large-capacity storage device such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for processing by the processing unit 11 . In the present embodiment, the storage unit 12 stores a program 12a executed by the processing unit 11. FIG. The storage unit 12 is also provided with a learning model storage unit 12b that stores information on a learned learning model that has undergone machine learning in advance.

In the present embodiment, the program (computer program, program product) 12a is provided in a form recorded in a recording medium 99 such as a memory card or an optical disk, and the information processing apparatus 1 reads out the program 12a from the recording medium 99 and stores it in the storage unit. 12. However, the program 12a may be written in the storage unit 12 at the manufacturing stage of the information processing device 1, for example. Further, for example, the program 12a may be distributed by a remote server device or the like and acquired by the information processing device 1 through communication. For example, the program 12 a may be recorded in the recording medium 99 and read by a writing device and written in the storage unit 12 of the information processing device 1 . The program 12 a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 99 .

The learning model storage unit 12b stores information about one or more learning models that the information processing apparatus 1 uses for processing such as abnormality detection. The information about the learning model can include, for example, information indicating the structure of the learning model, information about parameters of the learning model determined by machine learning processing, and the like. In the present embodiment, a learning model that performs segmentation processing for identifying pixels in the captured image of the camera 3 in which the object 100 is captured, and a class classification process that identifies the type of the target captured in the captured image. Information such as a learning model to be performed and/or a learning model for determining the presence or absence of an abnormality based on an image area in which the object 100 is photographed is stored in the learning model storage unit 12b.

The communication unit 13 is connected to the camera 3 and the display device 5 via, for example, a communication cable, and communicates with the camera 3 and the display device 5 . The communication unit 13 may perform communication based on a communication standard such as USB (Universal Serial Bus) or LAN (Local Area Network). Alternatively, the communication unit 13 may communicate with the camera 3 and the display device 5 by wireless communication without using a communication cable. The communication unit 13 receives, for example, a visible light image, an infrared image, and the like transmitted from the camera 3 , stores them in the storage unit 12 , and transmits control instructions and the like given from the processing unit 11 to the camera 3 . Further, the communication unit 13 transmits image data for display given from the processing unit 11 to the display device 5, and causes the display device 5 to display an image.

The operation unit 14 receives a user's operation and notifies the processing unit 11 of the received operation. The operation unit 14 receives a user's operation using, for example, mechanical buttons or an input device such as a mouse or keyboard. These input devices may be detachable from the information processing apparatus 1 . Further, the information processing device 1 does not have to include the operation unit 14 . The information processing device 1 may receive a user's operation through an operation unit provided in the display device 5, for example.

The display device 5 is, for example, a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display device 5 is connected to the information processing device 1 via a communication line, a signal line, or the like, and displays various images, characters, and the like given from the information processing device 1 .

Further, in the information processing apparatus 1 according to the present embodiment, the program 12a stored in the storage unit 12 is read out and executed by the processing unit 11, so that the photographed image acquisition unit 11a, the image area identification unit 11b, the abnormality detection unit 11c, the display processing unit 11d, and the like are implemented in the processing unit 11 as software functional units. In addition, in this figure, as the functional units of the processing unit 11, the functional units related to the abnormality detection processing of the target object 100 are illustrated, and the functional units related to other processing are omitted.

The photographed image acquisition unit 11a transmits and receives data to and from the camera 3 through the communication unit 13, thereby obtaining a photographed image of the object 100 photographed by the camera 3, that is, a visible light image photographed in the visible light wavelength band. and an infrared image taken in the infrared wavelength band. The captured image acquisition unit 11 a temporarily stores the acquired visible light image and infrared image in the storage unit 12 . The photographed image acquisition unit 11a may also perform controls related to photographing by the camera 3, such as shutter speed change, brightness adjustment, and focusing.

Based on the visible light image acquired by the captured image acquisition unit 11a from the camera 3, the image region specifying unit 11b uses a learning model for performing segmentation processing stored in the learning model storage unit 12b, and identifies an object from the visible light image. A process of specifying an image area in which the object 100 is photographed is performed. The learning model that performs the segmentation process is a learning model that has undergone machine learning in advance so as to receive, for example, a captured image as an input and output information indicating whether each pixel of the captured image belongs to the target object 100. be. A learning model such as CNN (Convolutional Neural Network), FCN (Fully Convolutional Networks), SegNet, or U-Net can be adopted as a learning model for segmentation processing. Since the learning model that performs the segmentation process is an existing technique, the detailed configuration, machine learning method, and the like will not be described.

Further, the image area specifying unit 11b performs processing for specifying an image area in which the object 100 is captured in the infrared image based on the result of specifying the position of the object 100 in the visible light image. In this embodiment, it is assumed that the visible light image and the infrared image captured by the camera 3 show the object 100 at the same position. Therefore, in the present embodiment, the image area specifying unit 11b can use the result of specifying the position of the target object 100 in the visible light image as the result of specifying the position of the target object 100 in the infrared image. However, if there is a difference in the position of the target object 100 between the visible light image and the infrared image captured by the camera 3, the image area specifying unit 11b corrects the position according to this difference to the specified result of the visible light image. By doing so, it is possible to obtain the result of specifying the position of the target object 100 in the infrared image.

The abnormality detection unit 11c extracts from the infrared image an image area in which the object 100 identified by the image area identification unit 11b is photographed, and based on the extracted image area, determines whether or not there is an abnormality or an abnormal location with respect to the object 100. Perform detection processing. In the present embodiment, the abnormality detection unit 11 c performs abnormality detection processing using one or more learning models stored in the learning model storage unit 12 b of the storage unit 12 . For example, the learning model is a learning model that has undergone machine learning in advance so as to receive an input of an image area extracted from an infrared image and output whether or not there is an abnormality in the object 100 captured in this image area. . Further, for example, the learning model receives input of an image area, detects a foreign object contained in the target object 100 photographed in this image area, and outputs information of a rectangular frame (so-called bounding box) surrounding the detected foreign object. It may be a learning model that has undergone machine learning in advance. Further, for example, the learning model receives input of an image region, detects a foreign object contained in the object 100 photographed in this image region, and specifies pixels in the image corresponding to the detected foreign object (performs so-called segmentation). ), it may be a learning model that has undergone machine learning in advance. Any learning model other than these may be used by the abnormality detection unit 11c. The abnormality detection unit 11c inputs information of an image region extracted from the infrared image to the learning model, acquires information output by the learning model, and detects an abnormality of the target object 100 based on the acquired information.

The learning model used by the abnormality detection unit 11c includes, for example, an image area of the object 100 extracted from an infrared image taken in an infrared wavelength band, and whether or not the object 100 photographed in this image area contains an abnormality. is generated in advance by a so-called supervised machine learning method using learning data associated with information indicating . For the learning model, for example, a learning model having a configuration such as a CNN (Convolutional Neural Network) can be adopted. In the present embodiment, the information processing device 1 performs learning processing for generating a learning model, but the present invention is not limited to this, and a device different from the information processing device 1 generates a learning model, The information processing apparatus 1 may acquire the information of the learned learning model from this device, store it in the learning model storage section 12b of the storage section 12, and use it for the abnormality detection process.

The display processing unit 11 d performs processing for displaying various information on the display device 5 by transmitting various information such as images and characters to the display device 5 . In the present embodiment, the display processing unit 11d displays information on the result of abnormality detection by the abnormality detection unit 11c on the display device 5. FIG. The display processing unit 11d displays, on the display device 5, a message notifying that, for example, a foreign substance contained in the object 100 to be inspected or an abnormality of the object 100 is detected. In addition, the display processing unit 11d displays a photographed image (visible light image or infrared image) of the object 100 in which the foreign matter or abnormality is detected, and also displays a symbol or figure indicating the location of the detected foreign matter or abnormality. It is superimposed and displayed.

FIG. 3 is a schematic diagram showing the configuration of the camera 3 according to this embodiment. The camera 3 according to this embodiment includes a control section 31, an imaging section 32, a switchable filter 33, a switching section 34, an infrared LED (Light Emitting Diode) 35, a communication section 36, and the like. The control unit 31 is configured using an arithmetic processing unit such as a CPU or MPU, a ROM, a RAM, and the like. The control unit 31 controls the operation of each unit included in the camera 3, and performs various processes such as photographing the object 100, switching filters, and transmitting image data obtained by photographing.

The imaging unit 32 has an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and an optical element such as a lens that converges light on the imaging element. In addition, the imaging device included in the photographing unit 32 according to the present embodiment has sensitivity not only to visible rays but also to infrared rays. The imaging unit 32 performs imaging through the first filter or the second filter of the switchable filter 33 and outputs image data obtained by imaging to the control unit 31 .

The switchable filter 33 has two optical filters, a first filter and a second filter. The switchable filter 33 can move the positions of the first filter and the second filter by, for example, a slide mechanism or a rotation mechanism. By this movement, the switchable filter 33 can switch between imaging by the imaging unit 32 via the first filter and imaging via the second filter. In the present embodiment, the first filter included in the switchable filter 33 transmits light in the wavelength band of the visible light region (eg, 380 nm to 780 nm) and does not transmit light in the wavelength band of the infrared region (eg, 780 nm to 1000 μm). . The second filter transmits light in the infrared wavelength band and does not transmit light in the visible light wavelength band. Accordingly, the photographing unit 32 can switch between photographing with visible light through the first filter and photographing with infrared light through the second filter.

The switching unit 34 switches the positions of the first filter and the second filter included in the switchable filter 33 by controlling the operation of a driving mechanism such as an actuator or a motor. For example, when the camera 3 operates in two modes, that is, a first mode for photographing with visible rays and a second mode for photographing with infrared rays, the switching unit 34 receives information indicating which mode is the current mode. Input from the control unit 31 . The switching unit 34 operates the driving mechanism according to the mode information input from the control unit 31 to switch the switching filter 33 .

The infrared LED 35 is a light-emitting element that emits light in a wavelength band in the infrared region. The infrared LED 35 is turned on and off under the control of the controller 31 . In this embodiment, the infrared LED 35 is turned on when the camera 3 performs infrared photography, and the infrared LED 35 is turned off when the camera 3 performs visible light photography.

Note that the infrared LED 35 may not be composed of one type of light emitting element, but may be composed of a plurality of light emitting elements that emit infrared rays in different wavelength bands, for example. The camera 3 may cause a plurality of light emitting elements to emit light simultaneously, for example, or may be switched to emit light individually, for example. Further, the camera 3 uses a filter that transmits infrared rays in a specific wavelength band, and limits the infrared rays in a wide wavelength band emitted by the infrared LED 35 to infrared rays in a narrow wavelength band to irradiate the object 100. good. The camera 3 may include a plurality of such filters and switch them as appropriate. Further, the camera 3 may include an infrared laser, for example, instead of the infrared LED 35, and may further include an optical element such as a lens that magnifies the light emitted by the infrared laser and irradiates the object 100 with the light. Also, the camera 3 may not have a mechanism for emitting infrared rays such as the infrared LED 35 .

The communication unit 36 is connected to the information processing device 1 via a communication cable, for example, and communicates with the information processing device 1 . The communication unit 36 may perform communication based on a communication standard such as USB or LAN, for example. Alternatively, the communication unit 36 may communicate with the information processing device 1 by wireless communication without using a communication cable. The communication unit 36 receives, for example, a command or the like transmitted from the information processing device 1 and provides it to the control unit 31, and also processes data provided from the control unit 31 (for example, data of an image captured by the imaging unit 32). Send to device 1.

<Status detection processing>
In the information processing system according to the present embodiment, for example, when the user who performs inspection work places the object 100 on the inspection table 7 and performs an inspection start operation on the operation unit 14 of the information processing apparatus 1, A state detection process of the object 100 is started. The information processing apparatus 1 transmits an instruction to photograph the object 100 to the camera 3, and the camera 3 photographs the object 100 in response to this instruction. The camera 3 first captures an image of the target object 100 using visible light by capturing an image of the target object 100 using the imaging unit 32 through the first filter of the switchable filter 33 . Next, the camera 3 switches the switchable filter 33 by the switching unit 34, turns on the infrared LED 35, and shoots the object 100 by the imaging unit 32 via the second filter. to shoot. The camera 3 transmits a visible light image and an infrared image of the object 100 obtained by photographing to the information processing device 1 .

FIG. 4 is an image showing an example of a visible light image and an infrared image of the object 100 photographed by the camera 3 . The upper part of FIG. 4 shows an example of a visible light image of two avocados taken as the object 100, and the lower part of FIG. 4 shows an example of an infrared image of the same object 100 taken. In the visible light image, the surface of the avocado is photographed with a dark color, so it is difficult for a human to visually determine whether there are any abnormalities or not, and it is also difficult to make a determination using a machine-learning model.

An infrared image, on the other hand, shows the surface of the avocado in bright colors. The avocado on the right in the exemplary infrared image has a uniformly bright color on the surface, whereas the avocado on the left has a mottled dark pattern on the surface. In this example, the avocado on the right is normal (fresh, not rotten) and the avocado on the left is abnormal (ripe, rotten). The black patterned portions on the surface of the avocado on the left are abnormal portions (ripe portions and rotten portions) of the avocado. In this way, with an infrared image, it is possible to determine the presence or absence of an abnormality, etc., by observing the difference in color on the surface of a dark-colored food such as avocado. Abnormality determination using a machine-learned learning model based on an infrared image of an avocado can be expected to obtain an easy and highly accurate result compared to the case of using a visible light image.

However, for example, when processing such as detection or identification of the object 100 captured in the captured image is performed using a learning model based on machine learning, the presence or absence of color of the object 100 may affect processing accuracy. Therefore, in the information processing system according to the present embodiment, the camera 3 captures two types of images, a visible light image and an infrared image. is detected (segmentation), and the presence or absence of an abnormality is determined based on the infrared image.

FIG. 5 is a schematic diagram for explaining the segmentation processing performed by the information processing device 1. As shown in FIG. The information processing apparatus 1 according to the present embodiment acquires two types of captured images, a visible light image captured by the camera 3 of the object 100 using visible light, and an infrared image captured using infrared rays. The information processing apparatus 1 is provided with a learning model (segmentation model) that performs machine learning in advance and performs segmentation processing, and inputs a visible light image acquired from the camera 3 to the segmentation model.

A segmentation model is a learning model that labels each pixel of an input image. In this embodiment, the segmentation model is machine-learned in advance so as to perform binary labeling indicating whether or not the object 100 (avocado in this example) is shown in each pixel of the input image. Learning model. When the input image has a size of M pixels×N pixels, for example, the information output by the segmentation model is an M×N array (vector, matrix), and this information is called mask data in the present embodiment. The mask data in the M×N array contains label values assigned to pixels at corresponding positions in the input image (for example, label "1" for pixels where the object 100 is shown, A label "0") is set for the pixel.

An example of mask data output by the segmentation model when the visible light image in the upper part of FIG. 5 is input to the segmentation model is shown in the lower part of FIG. The mask data in the lower part of FIG. The mask data is displayed as an image. The information processing device 1 inputs the visible light image acquired from the camera 3 to the segmentation model, and acquires the mask data output by the segmentation model.

Next, the information processing apparatus 1 according to the present embodiment uses the mask data acquired based on the visible light image to extract an image area in which the object 100 is captured from the infrared image acquired together with the visible light image. Perform extraction processing. FIG. 6 is a schematic diagram for explaining image region extraction processing performed by the information processing apparatus 1. As shown in FIG. The upper part of FIG. 6 shows the same mask data as the lower part of FIG. In the information processing system according to the present embodiment, the visible light image and the infrared image of the object 100 photographed by the camera 3 show the object 100 at the same position. The information processing apparatus 1 extracts the pixels of the infrared image corresponding to the positions labeled to indicate that the object 100 is captured, based on the mask data acquired based on the visible light image. , an image region in which the object 100 is shown can be extracted from the infrared image. The lower part of FIG. 6 shows the image area of the object 100 (avocado) extracted from the infrared image based on the mask data.

Note that the image area of the object 100 extracted from the infrared image can have various sizes, shapes, and the like. For this reason, the information processing apparatus 1, for example, replaces the pixel values of the pixels in the infrared image in which the object 100 is not shown with a predetermined value (for example, "0") as the extraction result of the object 100 from the infrared image. It may be used in subsequent processing. For example, when using mask data in which the label "1" is set for pixels in which the object 100 is captured and the label "0" is set for pixels in which the target object 100 is not captured, the information processing apparatus 1 uses each of the infrared images. By calculating the product of the pixel value of the pixel and the value of the corresponding mask data, data obtained by extracting the image area of the object 100 from the infrared image can be obtained.

Next, the information processing apparatus 1 according to the present embodiment performs processing for detecting abnormality of the object 100 based on the image area of the object 100 extracted from the infrared image. FIG. 7 is a schematic diagram for explaining anomaly detection processing performed by the information processing device 1. As shown in FIG. Note that the upper part of FIG. 7 shows the image area of the object 100 extracted from the same infrared image as the lower part of FIG. The information processing apparatus 1 according to the present embodiment includes a learning model (abnormality detection model) for detecting anomalies in the target object 100 for which machine learning has been performed in advance. The information processing apparatus 1 extracts the image area of the object 100 from the infrared image based on the mask data, and inputs data of the extracted image area to the abnormality detection model.

The anomaly detection model according to the present embodiment identifies an anomalous portion of the object 100 depicted in the input image, and outputs information such as the position and size of the identified anomalous portion. Further, the anomaly detection model according to the present embodiment is machine-learned in advance so that each pixel of an input image (image region) is labeled with a binary value indicating whether or not it is an anomaly. learning model. In the lower part of FIG. 7, the image area of the object 100 input to the anomaly detection model is superimposed with a thick line surrounding an anomaly location specified by the anomaly detection model, which is shown as output information of the anomaly detection model. . According to the result of this abnormality detection, it can be seen that the object 100 on the right side of the two objects 100 has an abnormal portion, and the object 100 on the left side does not have an abnormal portion. For example, the abnormality detection model may be configured to determine whether or not there is an abnormality in the object 100 and output binary information indicating the presence or absence of an abnormality or a numerical value indicating the possibility of an abnormality.

The information processing apparatus 1 inputs the image area of the target object 100 extracted from the infrared image to the abnormality detection model, and acquires the identification result of the abnormality location output by the abnormality detection model. The information processing device 1 displays on the display device 5 a screen for notifying the user of the location of the abnormality based on the acquired identification result. FIG. 8 is a schematic diagram showing an example of a notification screen of an abnormal location. The information processing apparatus 1 according to the present embodiment displays, on the display device 5, a thick line surrounding an abnormal portion of the object 100 specified by the abnormality detection model, for example, superimposed on the corresponding visible light image. For this purpose, the information processing device 1 transmits to the display device 5 information such as a visible light image to be displayed and position information of a thick line superimposed on the visible light image.

In this example, the information processing apparatus 1 superimposes a thick line surrounding the abnormal portion on the visible light image and displays it. However, the present invention is not limited to this. may Further, the information processing apparatus 1 may display a visible light image superimposed with a thick line surrounding an abnormal location and an infrared image superimposed with a thick line surrounding an abnormal location side by side. Further, the display of the state detection result by the information processing device 1 is not limited to the display surrounding the abnormal portion with a thick line, and may be displayed in various forms. In addition, the information processing apparatus 1 performs abnormality detection of the target object 100 from the infrared image using an abnormality detection model machine-learned in advance, but the present invention is not limited to this. The information processing apparatus 1 may detect pixels darker than a predetermined threshold value as abnormal locations, for example, based on the shading of each pixel in the infrared image.

Instead of displaying the result of abnormality detection on the display device 5, the information processing apparatus 1 may notify the user of the result of abnormality detection by outputting, for example, an electronic sound or a voice message from a speaker. The information processing device 1 may notify the user of the result of abnormality detection by any method.

FIG. 9 is a flow chart showing the procedure of state detection processing performed by the information processing apparatus 1 according to the present embodiment. The captured image acquisition unit 11a of the processing unit 11 of the information processing apparatus 1 according to the present embodiment causes the camera 3 to capture an image of the object 100 placed on the inspection table 7 according to the user's operation, for example. 3 obtains a visible light image of the object 100 photographed with visible light and an infrared image of the object 100 photographed with infrared light (step S1).

Next, the image region specifying unit 11b of the processing unit 11 inputs the visible light image among the images acquired in step S1 to the segmentation model that has undergone machine learning in advance (step S2). In response to the input of the visible light image, the segmentation model outputs mask data labeled as to whether or not the object 100 is shown in each pixel of the visible light image. The image region specifying unit 11b acquires this mask data output by the segmentation model (step S3). Based on the mask data acquired in step S3, the image area specifying unit 11b extracts an image area in which the object 100 is shown from the infrared image acquired in step S1 (step S4).

Next, the abnormality detection unit 11c of the processing unit 11 inputs the data of the image region of the target object 100 extracted in step S4 to an abnormality detection model that has undergone machine learning in advance (step S5). According to the input of the image area, the abnormality detection model identifies an abnormal location of the object 100 captured in the image area, and outputs information such as the position and size of the identified abnormal location as an abnormality detection result. The abnormality detection unit 11c acquires the abnormality detection result output by the abnormality detection model (step S6). The abnormality detection unit 11c determines whether or not there is an abnormality in the object 100 based on the abnormality detection result obtained in step S6 (step S7). If there is no abnormality in the object 100 (S7: NO), the abnormality detection section 11c terminates the process.

If there is an abnormality in the object 100 (S7: YES), the display processing unit 11d of the processing unit 11 displays, for example, the visible light image acquired in step S1 and the abnormality detection result acquired in step S6 on the display device 5. , the display device 5 displays an image in which a thick line surrounding the detected abnormal portion is superimposed on the visible light image of the object 100 (step S8), and the process ends.

<Summary>
In the information processing system according to the present embodiment configured as described above, the information processing apparatus 1 captures the first image of the target object 100 with the camera 3 based on the light (visible light) in the first wavelength band (first characteristic). (visible light image) is acquired, and a second image (infrared image) of the object 100 captured by the camera 3 based on the light (infrared rays) of the second wavelength band (second characteristic) is acquired. The information processing apparatus 1 performs segmentation on the visible light image to identify an image region in which the object 100 is captured, and detects the state of the object 100 from the infrared image based on the identification result. As a result, the information processing system can be expected to perform highly accurate segmentation processing using the visible light image, and can be expected to perform highly accurate state detection processing using the infrared image.

In the present embodiment, the wavelength band of visible light is used as the first wavelength band, and the wavelength band of infrared rays is used as the second wavelength. may be appropriately selected according to the type of the object 100 or the like. For example, an ultraviolet wavelength band or an X-ray wavelength band may be used, or a red light wavelength band or a blue light wavelength band among visible rays may be used. The combination of the first wavelength band and the second wavelength band may also be selected as appropriate. For example, the first wavelength band may be an infrared wavelength band, and the second wavelength band may be a visible light wavelength band. Wavebands other than the wavelength bands of may be appropriately combined. Further, for example, the first wavelength band may be the wavelength band of visible light, and the second wavelength band may be the wavelength band of blue light in the visible light. good too.

Further, the information processing apparatus 1 not only performs segmentation on the first image (visible light image), but also detects the state of the object 100 based on the first image and detects the state of the object 100 based on the second image (infrared image). It may be done in conjunction with detection. Whether or not to perform state detection based on the first image can be selected depending on the type of the target object 100, for example.

In the present embodiment, the camera 3 captures images in two types of wavelength bands, and the information processing apparatus 1 obtains two types of captured images of the object 100. However, the present invention is not limited to this. , three or more types of captured images of the object 100 captured by the camera 3 in three or more wavelength bands may be obtained. For example, when acquiring three types of photographed images, the information processing apparatus 1 identifies an image region in which the target object 100 is captured based on the first image, and based on the identification result, selects the image area from the second image and the third image. State detection can be performed by extracting an image region in which the object 100 is captured. Further, for example, the information processing device 1 identifies an image region in which the object 100 is shown based on the first image, and then identifies an image region in which the object 100 is shown from a synthesized image obtained by synthesizing the second image and the third image. may be extracted. Further, for example, the information processing device 1 may specify an image region in which the target object is captured based on a synthesized image obtained by synthesizing the first image and the second image. The information processing apparatus 1 appropriately utilizes a plurality of captured images respectively captured by the camera 3 in a plurality of wavelength bands to identify an image region in which the target object 100 is captured, and to identify the image region from the captured image based on the identification result. and state detection of the object 100 based on the extracted image area.

In addition, the information processing system according to the present embodiment detects an abnormality of the object 100 as the state detection of the object 100, but is not limited to this. , presence or absence of foreign matter, etc. may be detected. In addition, although avocado is exemplified as the object 100, the object 100 may be various foods such as apples, blueberries, strawberries, oranges, etc., or may be various articles other than foods, and may be human or human. It may be various animals such as pets, or anything else.

In addition, in the information processing system according to the present embodiment, one camera 3 is configured to capture images in a plurality of wavelength bands, but the present invention is not limited to this. You can take pictures. For example, a first camera that captures images with visible light and a second camera that captures images with infrared rays may be arranged side by side, and the object 100 may be captured simultaneously by the first camera and the second camera. . For example, when the object 100 is moving on a belt conveyor, the first camera and the second camera are arranged side by side in the moving direction of the belt conveyor, and the first camera and the second camera are arranged with a time difference in consideration of the moving speed of the belt conveyor. By photographing, the object 100 can be photographed at approximately the same position in the first photographed image by the first camera and the second photographed image by the second camera.

In addition, in the information processing system according to the present embodiment, a plurality of photographed images photographed based on a plurality of lights having different wavelength bands are used as light characteristics. For example, a plurality of captured images captured based on a plurality of lights with different polarization directions may be used. For example, one of a plurality of filters included in the switchable filter 33 of the camera 3 may be a polarizing filter, and the camera 3 may perform both imaging using visible light and imaging via the polarizing filter. The switchable filter 33 of the camera 3 may be switchably provided with a plurality of polarizing filters having different directions, and a filter that transmits light of a predetermined wavelength and a polarizing filter that transmits light of a predetermined direction are mixed. may be The information processing device 1 can specify the image area of the object 100 based on the visible light image, for example, and detect the state of the object 100 based on the polarized image captured through the polarizing filter.

<Embodiment 2>
The information processing system according to the second embodiment detects the state of each object when a plurality of types of objects are photographed by the camera 3 as the target object 100 and a plurality of types of objects are mixed in the photographed image. It is a system that can The information processing device 1 of the information processing system according to the second embodiment uses a learning model that has undergone machine learning in advance so as to determine the type of each object 100 captured in the visible light image, that is, a classification model that performs so-called class classification. have. Based on the visible light image acquired from the camera 3, the information processing device 1 uses a segmentation model to specify an image region in which the target object 100 is captured, and uses a classification model to identify each image captured in the visible light image. Classify the type of object 100 . The information processing apparatus 1 may perform either of segmentation and class classification first, or may perform these two processes at the same time. Note that the information processing apparatus 1 may have a learning model that performs segmentation and class classification instead of having the segmentation model and the classification model separately. In this case, the learning model outputs, for each pixel of the visible light image, data such as an array with a label value indicating the type of the object 100 captured in this pixel as a result of segmentation and class classification. do.

Next, the information processing device 1 extracts an image region in which the object 100 is captured from the infrared image based on the results of segmentation and class classification of the object 100 based on the visible light image. The information processing apparatus 1 according to Embodiment 2 extracts an image region for each type of the object 100 based on the result of class classification. For example, if three fruits are photographed in the visible light image, two of which are apples and the remaining one is a mandarin orange, the information processing device 1 detects an image region in which two apples are photographed from the infrared image, An image region in which one mandarin orange is photographed is extracted.

Further, the information processing apparatus 1 according to Embodiment 2 has a plurality of abnormality detection models for performing abnormality detection for each type of the object 100 . The information processing device 1 selects one or a plurality of abnormality detection models according to the classification result from among the plurality of abnormality detection models, according to the classification result of the target object 100 based on the visible light image. The information processing apparatus 1 inputs the image regions extracted for each type of the target object 100 from the infrared image to the corresponding abnormality detection models, and acquires the abnormality detection results output by each abnormality detection model. The information processing device 1 integrates the abnormality detection results for each type of the object 100 and displays on the display device 5 an image in which a thick line or the like surrounding the abnormal portion is superimposed on the visible light image.

Further, the information processing apparatus 1 according to the second embodiment detects an abnormality of the target object 100 from the infrared image using an abnormality detection model machine-learned in advance, but the present invention is not limited to this. The information processing apparatus 1 may detect pixels darker than a predetermined threshold value as abnormal locations, for example, based on the shading of each pixel in the infrared image. In this case, the information processing apparatus 1 stores a threshold value for comparing shades for each type of the target object 100, and can select a threshold value according to the classification result of the target object 100 to perform anomaly detection. .

FIG. 10 is a flowchart showing the procedure of state detection processing performed by the information processing apparatus 1 according to the second embodiment. The captured image acquisition unit 11a of the processing unit 11 of the information processing apparatus 1 according to the second embodiment acquires a visible light image of the object 100 placed on the inspection table 7 captured by the camera 3 using visible light and an infrared image. Then, an infrared image is acquired (step S21). The image region specifying unit 11b of the processing unit 11 inputs the visible light image acquired in step S21 to the segmentation model that has undergone machine learning in advance (step S22). The image region identifying unit 11b acquires mask data output by the segmentation model as a result of segmentation processing (step S23).

The image region specifying unit 11b inputs the visible light image acquired in step S21 to a classification model that has undergone machine learning in advance (step S24). The classification model is a learning model that has undergone machine learning in advance so as to accept a photographed image of the object 100 as an input and output classification results such as the type of the object 100 . The image region specifying unit 11b acquires the classification results output by the classification model (step S25). Based on the mask data acquired in step S23, the image area specifying unit 11b extracts an image area in which the object 100 is shown from the infrared image acquired in step S21 (step S26).

Next, the abnormality detection unit 11c of the processing unit 11 selects an abnormality detection model for detecting an abnormality of the object 100 according to the classification result of the object 100 acquired in step S25 (step S27). The abnormality detection unit 11c inputs the data of the image area of the object 100 extracted in step S26 to the abnormality detection model selected in step S27 (step S28). The abnormality detection unit 11c acquires the abnormality detection result output by the abnormality detection model (step S29). The abnormality detection unit 11c determines whether or not there is an abnormality in the object 100 based on the abnormality detection result obtained in step S29 (step S30). If there is no abnormality in the object 100 (S30: NO), the abnormality detection section 11c terminates the process. If there is an abnormality in the object 100 (S30: YES), the display processing unit 11d of the processing unit 11 displays, for example, the visible light image acquired in step S21 and the abnormality detection result acquired in step S29 on the display device 5. , the display device 5 displays an image in which a thick line surrounding the detected abnormal portion is superimposed on the visible light image of the object 100 (step S31), and the process ends.

In the information processing system according to Embodiment 2 having the above configuration, the information processing device 1 performs class classification of the target object 100 based on the visible light image captured by the camera 3 . The information processing apparatus 1 selects a condition (for example, which of a plurality of anomaly detection models to use, or which of a plurality of thresholds to use, etc.) for anomaly detection of the target object 100 based on the classification result. . The information processing apparatus 1 selects one from a plurality of anomaly detection models provided for each type of the target object 100 based on the classification result, for example, and assigns the image area of the target object 100 extracted from the infrared image to the selected anomaly detection model. is input and the abnormality detection result output by the abnormality detection model is obtained, thereby performing abnormality detection of the target object 100 . As a result, the information processing system can be expected to accurately detect an abnormality based on the image captured by the camera 3 even when a plurality of types of objects are mixed.

In the information processing system according to Embodiment 2, the object 100 is classified into classes based on the visible light image captured by the camera 3. However, the present invention is not limited to this, and class classification is performed based on the infrared image. you can go

Further, other configurations of the information processing system according to the second embodiment are the same as those of the information processing system according to the first embodiment, so the same parts are denoted by the same reference numerals, and detailed description thereof will be omitted.

<Embodiment 3>
In the information processing system according to Embodiment 3, the camera 3 is capable of capturing images using visible light and capturing images using a plurality of infrared rays having different wavelength bands. In the information processing system according to Embodiment 3, instead of the camera 3 first capturing a visible light image and an infrared image and transmitting both images to the information processing apparatus 1, the camera 3 first captures only the visible light image. to transmit only the visible light image to the information processing apparatus 1 . The information processing apparatus 1 according to the third embodiment performs class classification of the target object 100 based on the visible light image captured by the camera 3, and selects a wavelength band for capturing an infrared image according to the type of the classified target object 100. It decides and notifies the camera 3 of information on the decided wavelength band. For this reason, the information processing apparatus 1 is provided in advance with a table or the like that associates, for example, the type of the target object 100 with the wavelength band for capturing the infrared image.

The camera 3 has, for example, a plurality of filters that transmit infrared rays of different wavelength bands in the switchable filter 33 . The object 100 is photographed. The camera 3 transmits to the information processing apparatus 1 an infrared image of the object 100 photographed by infrared rays in the notified wavelength band. Note that when a plurality of objects 100 of different types are captured in the visible light image, the information processing apparatus 1 notifies the camera 3 of the plurality of wavelength bands, and the camera 3 captures images using infrared light in the plurality of wavelength bands. A plurality of infrared images may be transmitted to the information processing device 1 .

The information processing device 1 acquires an infrared image from the camera 3, and extracts an image region in which the target object 100 is captured from the infrared image based on the result of segmentation processing for the visible light image. The information processing device 1 selects an anomaly detection model according to the type of the target object 100, inputs the extracted image region to the anomaly detection model, and obtains an anomaly detection result output by the anomaly detection model, thereby obtaining an Anomaly detection of the object 100 can be performed.

FIG. 11 is a flowchart showing the procedure of state detection processing performed by the information processing apparatus 1 according to the third embodiment. The captured image acquisition unit 11a of the processing unit 11 of the information processing apparatus 1 according to the third embodiment acquires a visible light image of the object 100 placed on the inspection table 7 captured by the camera 3 using visible light (step S41). The image region specifying unit 11b of the processing unit 11 inputs the visible light image acquired in step S41 to the segmentation model that has undergone machine learning in advance (step S42). The image region specifying unit 11b acquires mask data output by the segmentation model as a result of segmentation processing (step S43).

The image area specifying unit 11b inputs the visible light image acquired in step S41 to a classification model that has undergone machine learning in advance (step S44). The image region specifying unit 11b acquires the classification result output by the classification model (step S45). The image region specifying unit 11b selects the infrared wavelength band corresponding to the classification result obtained in step S45 by referring to a table storing, for example, the correspondence between the type of the target object 100 and the infrared wavelength band ( step S46). The image area specifying unit 11b notifies the infrared wavelength band selected in step S46 to the camera 3 (step S47). The captured image acquiring unit 11a acquires the infrared image captured by the camera 3 in response to the notification of step S47 (step S48). Based on the mask data acquired in step S43, the image area specifying unit 11b extracts an image area in which the object 100 is shown from the infrared image acquired in step S48 (step S49).

Next, the abnormality detection unit 11c of the processing unit 11 selects an abnormality detection model for detecting an abnormality of the object 100 according to the classification result of the object 100 acquired in step S45 (step S50). The abnormality detection unit 11c inputs the data of the image area of the object 100 extracted in step S49 to the abnormality detection model selected in step S50 (step S51). The abnormality detection unit 11c acquires the abnormality detection result output by the abnormality detection model (step S52). The abnormality detection unit 11c determines whether or not there is an abnormality in the object 100 based on the abnormality detection result acquired in step S52 (step S53). If there is no abnormality in the object 100 (S53: NO), the abnormality detection section 11c terminates the process. If there is an abnormality in the object 100 (S53: YES), the display processing unit 11d of the processing unit 11 displays, for example, the visible light image acquired in step S41 and the abnormality detection result acquired in step S52 on the display device 5. , the display device 5 displays an image in which a thick line surrounding the detected abnormal portion is superimposed on the visible light image of the object 100 (step S54), and the process ends.

In the information processing system according to Embodiment 3 having the above configuration, the information processing device 1 classifies the target object 100 based on the visible light image acquired from the camera 3, and selects the infrared wavelength band according to the classification result. In response to this notification, the camera 3 performs photographing with infrared light in the selected wavelength band. The camera 3 transmits an infrared image of the object 100 photographed with infrared rays in the wavelength band corresponding to the notification to the information processing device 1, and the information processing device 1 detects an abnormality of the object 100 based on the infrared image acquired from the camera 3. I do. As a result, in the information processing system according to the third embodiment, the camera 3 can capture an infrared ray in a wavelength band suitable for detecting anomalies such as the type of the target object 100 . Anomaly detection can be performed based on the infrared image, and improvement in the accuracy of anomaly detection can be expected.

In the third embodiment, the information processing apparatus 1 selects the infrared wavelength band according to the type of the target object 100, but it is not limited to this. For example, the camera 3 is configured to be capable of photographing by switching the wavelength band of visible light to a plurality of wavelength bands. may be selected to cause the camera 3 to photograph the object 100 .

Further, other configurations of the information processing system according to Embodiment 3 are the same as those of the information processing systems according to Embodiments 1 and 2. Therefore, similar portions are denoted by the same reference numerals, and detailed description thereof is omitted. do.

<Embodiment 4>
The learning model used by the information processing apparatus 1 that classifies the object 100 based on the visible light image outputs, as a classification result, a numerical value indicating the probability that the object 100 belongs to each class. . For example, when classifying whether the object 100 belongs to class A, B or C, the learning model has a confidence of 0.7 for class A, a confidence of 0.2 for class B, and a confidence of 0.2 for class C. Machine learning is performed in advance so as to output a numerical value of certainty for each class, such as 0.1. The information processing device 1 compares the confidence of each class and classifies the class corresponding to the largest value as the class of the object 100 .

The information processing apparatus 1 according to Embodiment 4 classifies the target object 100 into classes based on the visible light image acquired from the camera 3 using a learning model that has undergone machine learning in advance. The information processing apparatus 1 acquires the certainty factor of each class output by the learning model, and determines the class with the largest certainty factor value as the class of the object 100 . At this time, the information processing apparatus 1 according to Embodiment 4 compares the confidence in the class with the highest confidence and the class with the second highest confidence, and if the difference in confidence between the two classes is less than the threshold Then, the class classification process is performed again.

The information processing apparatus 1 according to Embodiment 4 stores information on infrared wavelength bands in advance in a table or the like in association with a combination of two classes that are classification targets of the object 100 . This information is information of wavelength bands that can be classified into two classes using an infrared image, and the wavelength bands are set in advance by the designer or the like of the information processing system according to the present embodiment. For example, if there are three classes A, B and C, the wavelength band for classifying the classes A and B, the wavelength band for classifying the classes B and C, and the classes C and A are classified. A wavelength band for and can be set in advance.

When the difference in confidence between the top two classes output by the learning model is less than the threshold, the information processing device 1 acquires the infrared wavelength bands associated with these two classes by referring to the above table. do. The information processing device 1 notifies the camera 3 of the acquired information on the infrared wavelength band. The camera 3 according to the fourth embodiment switches the filter according to the wavelength band notified from the information processing device 1 in the same manner as the camera 3 according to the third embodiment, and captures the object 100 with infrared rays. conduct. The camera 3 transmits to the information processing apparatus 1 an infrared image of the object 100 photographed by infrared rays in the notified wavelength band.

The information processing device 1 acquires the infrared image from the camera 3, and classifies the object 100 again based on the acquired infrared image. At this time, the learning model used by the information processing apparatus 1 for class classification is machine-learned in advance so as to perform class classification based on the infrared image, unlike the learning model that classifies the class based on the visible light image. is a model. The information processing device 1 acquires from the learning model the results of class reclassification based on the infrared image, selects an anomaly detection model based on the classification result, and generates the infrared image (the infrared image used for reclassification and (which may be the same or different) is input to the anomaly detection model, and the anomaly detection result output by the anomaly detection model is acquired to detect an anomaly of the object 100. I do.

12 and 13 are flowcharts showing the procedure of state detection processing performed by the information processing apparatus 1 according to the fourth embodiment. The captured image acquisition unit 11a of the processing unit 11 of the information processing apparatus 1 according to the fourth embodiment obtains a visible light image of the target object 100 captured by the camera 3 using visible light and an image of the target object 100 captured using infrared light. An infrared image is obtained (step S61). The image region specifying unit 11b of the processing unit 11 inputs the visible light image acquired in step S61 to the segmentation model that has undergone machine learning in advance (step S62). The image region specifying unit 11b acquires mask data output by the segmentation model as a result of segmentation processing (step S63).

Further, the image region specifying unit 11b inputs the visible light image acquired in step S61 to a classification model that has undergone machine learning in advance (step S64). The image region specifying unit 11b acquires the classification result output by the classification model (step S65). Based on the classification results obtained in step S65, the image region specifying unit 11b determines whether the difference in confidence between the top two classes is less than a threshold (step S66). If the difference in certainty is equal to or greater than the threshold (S66: NO), the image region specifying unit 11b determines the class with the highest certainty in the classification result acquired in step S65 as the type of the object 100, and proceeds to step S72. proceed.

If the confidence factor difference is less than the threshold (S66: YES), the image region specifying unit 11b refers to a table that stores, for example, combinations of types of the target object 100 and infrared wavelength bands, and performs step Infrared wavelength bands corresponding to the combination of the top two classes in the classification results obtained in S65 are selected (step S67). The image area specifying unit 11b notifies the camera 3 of the infrared wavelength band selected in step S67 (step S68). The captured image acquisition unit 11a acquires the infrared image captured by the camera 3 in response to the notification of step S68 (step S69). The image region specifying unit 11b inputs the infrared image acquired in step S69 to a classification model that has undergone machine learning in advance (step S70). The image region specifying unit 11b acquires the classification result output by the classification model (step S71), and advances the process to step S72.

Next, the image area specifying unit 11b extracts an image area including the object 100 from the infrared image acquired in step S61 based on the mask data acquired in step S63 (step S72). The abnormality detection unit 11c of the processing unit 11 selects an abnormality detection model for detecting an abnormality of the object 100 according to the classification result obtained in step S65 or the reclassification result obtained in step S71. (step S73). The abnormality detection unit 11c inputs the data of the image area of the object 100 extracted in step S72 to the abnormality detection model selected in step S73 (step S74). The abnormality detection unit 11c acquires the abnormality detection result output by the abnormality detection model (step S75).

The abnormality detection unit 11c determines whether or not there is an abnormality in the object 100 based on the abnormality detection result obtained in step S75 (step S76). If there is no abnormality in the object 100 (S76: NO), the abnormality detection section 11c terminates the process. If there is an abnormality in the object 100 (S76: YES), the display processing unit 11d of the processing unit 11 displays, for example, the visible light image acquired in step S61 and the abnormality detection result acquired in step S75 on the display device 5. , the display device 5 displays an image in which a thick line surrounding the detected abnormal portion is superimposed on the visible light image of the object 100 (step S77), and the process ends.

The information processing apparatus 1 according to Embodiment 4 having the above configuration classifies the object 100 into classes based on the visible light image, and when the confidence of classification is low for the top two classes (the confidence of the two classes When the difference is less than the threshold), an infrared wavelength band corresponding to these two classes is selected, and an infrared image obtained by imaging the object 100 based on the selected wavelength band by the camera 3 is acquired and acquired. Based on the infrared image, the object 100 is reclassified into classes. As a result, the information processing system can be expected to perform class classification based on the infrared image to improve accuracy when the degree of confidence in the class classification based on the visible light image is low.

Since other configurations of the information processing system according to Embodiment 4 are the same as those of the information processing systems according to Embodiments 1, 2, and 3, similar portions are denoted by the same reference numerals, and detailed descriptions thereof are omitted. omitted.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above-described meaning, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

The matters described in each embodiment can be combined with each other. Also, the independent claims and dependent claims described in the claims can be combined with each other in all possible combinations regardless of the form of citation. Furthermore, although the scope of claims uses a format in which claims refer to two or more other claims (multi-claim format), it is not limited to this. It may also be described using a format for describing multiple claims (multi-multiclaim) that refers to at least one multiple claim.

1 Information processing device (computer)
3 camera 5 display device 7 inspection table 11 processing unit 11a captured image acquisition unit (first acquisition unit, second acquisition unit)
11b Image area specifying unit (specifying unit)
11c Abnormality detection unit (detection unit)
11d display processing unit 12 storage unit 12a program (computer program)
12b learning model storage unit 13 communication unit 14 operation unit 31 control unit 32 imaging unit 33 switchable filter (first filter, second filter)
34 switching unit 35 infrared LED (light source)
36 communication unit 100 object

Claims (8)

The information processing device
Acquiring a first image of an object photographed based on light in a first wavelength band;
Acquiring a second image of the object photographed based on light in a second wavelength band;
segmenting the first image to identify an image region in which the object is captured;
extracting an image region of the object from the second image based on the identification result;
classifying the object based on the first image;
Selecting one learning model according to a classification result from among a plurality of learning models subjected to machine learning so as to output information regarding the state of the object in response to the input of the image region extracted from the second image. death,
detecting the state of the object based on the extracted image region and the selected learning model;
Information processing methods. The first wavelength band is a wavelength band of visible light,
The second wavelength band is an infrared wavelength band,
The information processing method according to claim 1 . The information processing device
selecting the second wavelength band according to the classification result;
Acquiring the second image taken based on the selected light in the second wavelength band;
The information processing method according to claim 1 . The information processing device
selecting the second wavelength band according to the two classes when the confidence in the classification of the object into two classes is low;
Acquiring the second image captured based on the selected light in the second wavelength band;
reclassifying the object based on the acquired second image;
The information processing method according to claim 1 . The information processing device
Acquiring the first image and the second image of a plurality of types of food,
Based on the identification result, classify each food type,
Detect the state of each food based on the classification results,
The information processing method according to claim 1 . to the computer,
Acquiring a first image of an object photographed based on light in a first wavelength band;
Acquiring a second image of the object photographed based on light in a second wavelength band;
segmenting the first image to identify an image region in which the object is captured;
extracting an image region of the object from the second image based on the identification result;
classifying the object based on the first image;
Selecting one learning model according to a classification result from among a plurality of learning models subjected to machine learning so as to output information regarding the state of the object in response to the input of the image region extracted from the second image. death,
detecting the state of the object based on the extracted image region and the selected learning model;
A computer program that causes a process to be performed. a first acquisition unit that acquires a first image of an object captured based on light in a first wavelength band;
a second acquisition unit that acquires a second image of the object captured based on light in a second wavelength band;
a specifying unit that performs segmentation on the first image to specify an image region in which the object is captured;
an extraction unit that extracts an image area of the object from the second image based on the identification result;
a classifying unit that classifies the object based on the first image;
Selecting one learning model according to a classification result from among a plurality of learning models subjected to machine learning so as to output information regarding the state of the object in response to the input of the image region extracted from the second image. a selection unit to
An information processing apparatus comprising: a detection unit that detects the state of the object based on the extracted image area and the selected learning model. a first filter that transmits light in a first wavelength band and blocks light in a second wavelength band; a second filter that transmits light in the second wavelength band and blocks light in the first wavelength band; A light source that emits light in two wavelength bands, an imaging unit that performs imaging via the first filter or the second filter, a switching unit that switches the imaging unit between the first filter and the second filter, and the imaging unit a photographing device having a control unit that causes the light source to emit light when photographing through the second filter;
a first acquisition unit that acquires a first image obtained by the imaging device capturing an object based on light in a first wavelength band; a second acquisition unit that acquires an image; an identification unit that performs segmentation on the first image to identify an image area in which the object is captured; and an image area of the object from the second image based on identification results. a classification unit for classifying the object based on the first image; machine learning to output information about the state of the object in response to input of the image region extracted from the second image a selection unit that selects one learning model according to a classification result from among the plurality of learning models subjected to the above; and a detection that detects the state of the object based on the extracted image region and the selected learning model. An information processing system comprising: an information processing device having a unit;

Images