JP7308436B2 - Information management device and information management method - Google Patents

Description

The present disclosure relates to an information management apparatus and an information management method that perform object confirmation processing using captured images.

2. Description of the Related Art Systems for performing various inspections, confirmations, judgments, etc. by image recognition processing using captured images captured by an imaging device are widely used. For example, in production facilities such as factories, confirmation and inspection are performed visually by workers, and confirmation, inspection, and the like are performed using captured images captured by an imaging device. Confirmation and inspection using photographed images include inspections such as quality judgment of products in the production process, as well as confirmation of molds, jigs, etc. used in the production process.

2. Description of the Related Art As a conventional technology that performs confirmation processing using a photographed image, for example, there is an article confirmation device and an article confirmation method disclosed in Japanese Patent Application Laid-Open No. 2002-200010. In this conventional example, image information indicating a characteristic image including unreproducible fine features of the surface of an article is acquired, and this image information is stored as characteristic information, and when determining whether the article is genuine or not. Then, image information in a predetermined area on the surface of the article is acquired as confirmation information, and pattern matching is performed between the acquired confirmation information and the stored characteristic information to determine whether or not the characteristic images are the same. If it is determined as such, the stored characteristic information is updated to the confirmation information. As a result, even if the surface of the article deteriorates, the authenticity of the article can be confirmed easily and with high accuracy.

JP 2007-041643 A

In confirmation processing using photographed images, the difference between the sample image used as the reference for verification and the confirmation target can vary depending on the difference in the shooting angle when photographing the confirmation target, the brightness of the subject, the amount of light of the lighting, and the irradiation angle of the light. There may be a case where it is not possible to appropriately determine the match with the photographed image.

Here, as an example of confirmation processing using a photographed image, a case is assumed in which collation is performed to determine whether or not a mold to be worked on is appropriate in a production process. In this case, for example, a photographed image of an appropriate mold is stored as a sample image, and an operator photographs the mold to be confirmed to be used, compares the photographed image with the sample image, and performs matching determination. methods are used. In particular, if the object to be checked is an object that reflects a lot of light from its surface, such as a metal plate with a curved surface, etc., the effect of the reflected light is also great, and depending on the shooting conditions of the object to be checked, the collation of the captured image may not be stable. may not be able to execute

The present disclosure has been devised in view of the above-described conventional circumstances, and is an information management apparatus and information management method capable of executing simple and appropriate collation processing when performing confirmation using a photographed image to be confirmed. intended to provide

The present disclosure includes a photographed image acquisition unit that acquires a photographed image of an article to be checked from a photographing device, a master image acquisition unit that acquires a master image that serves as a template for matching with the photographed image, and the photographed image. a control unit for executing matching processing of the photographed image by using a plurality of master images for the photographed image; and a storage unit for storing information. and (i) presenting the obtained plurality of captured images to the user when it is determined that the matching is unsuccessful. , accepting a selection of the photographed image as one of new master images, adding the selected photographed image to the plurality of master images, outputting a determination result of successful matching, and (ii) determining successful matching. In this case, the information management device updates the plurality of master images by replacing the photographed image that has been successfully collated with a master image determined to have been successfully collated among the plurality of master images.

Further, according to the present disclosure, a photographed image of an article to be confirmed is obtained from a photographing device, a master image serving as a template for matching with the photographed image is obtained, and a matching determination is made between the photographed image and the master image. is performed on the photographed image using a plurality of master images to perform collation processing of the photographed image, and in the collation processing, for the photographed image, the number of master images corresponding to the number of the plurality of master images (i) when it is determined that the matching has failed, the plurality of acquired captured images are presented to the user, the captured image is accepted as one of new master images, and the selected captured image is selected; to the plurality of master images, and outputting a determination result of successful matching; An information management method is provided for updating the plurality of master images by replacing the master images .

Further, according to the present disclosure, a photographed image of an article to be confirmed is obtained from a photographing device, a master image serving as a template for matching with the photographed image is obtained, and a matching determination is made between the photographed image and the master image. is performed on the photographed image using a plurality of master images to perform collation processing of the photographed image, and in the collation processing, for the photographed image, the number of master images corresponding to the number of the plurality of master images (i) when it is determined that the matching has failed, the user is presented with the acquired plurality of captured images, and the selected captured image is accepted as one of the new master images; to the plurality of master images, and outputting a determination result of successful matching; A program is provided that causes a computer to perform a process of updating the plurality of master images by replacing the master image .

Advantageous Effects of Invention According to the present disclosure, it is possible to perform simple and appropriate matching processing when performing confirmation using a photographed image to be confirmed.

1 is a block diagram showing a configuration example of an information management system including an information management device according to an embodiment; FIG. The figure which shows an example of the equipment information in embodiment A diagram showing examples of photographed images in a plurality of different modes when the mold is photographed. 4 is a flowchart showing an operation example of confirmation processing using a photographed image by the information management device according to Embodiment 1; 10 is a flowchart showing an operation example of confirmation processing using a photographed image by the information management device according to Embodiment 2; 10 is a flow chart showing an operation example of confirmation processing using a photographed image by the information management device according to the third embodiment; FIG. 4 is a diagram showing an example of a captured image selection screen as a first example of a display screen by the information management device; 14 is a flow chart showing an operation example of confirmation processing using a photographed image by the information management device according to the fourth embodiment; 14 is a flow chart showing an operation example of confirmation processing using a photographed image by the information management device according to the fifth embodiment; FIG. 11 is a diagram showing an example of a master image confirmation screen as a second example of the display screen by the information management device;

Hereinafter, each embodiment specifically disclosing the configuration according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for a thorough understanding of the present disclosure by those skilled in the art and are not intended to limit the claimed subject matter.

In the present embodiment, an information management system in production equipment such as a factory is exemplified as a system that performs confirmation processing using a photographed image. indicates An example of the configuration and operation of an information management device for verifying whether or not a mold to be worked on in a production process is appropriate will be described below.

FIG. 1 is a block diagram showing a configuration example of an information management system including an information management device according to an embodiment. The information management system 10 includes a management device 20 and terminal devices 30 . The management device 20 is configured by a management computer located at a central management base of an organization that operates the system, or a server device on a network, and implements each function. The management device 20 functions as a server device that performs various types of management related to production equipment.

The management device 20 has a control section 21 , a storage section 22 and a communication section 23 . The control unit 21 has a processor, and realizes various functions by executing a predetermined program held in the storage unit 22 by the processor, for example. The processor may include an MPU (Micro processing Unit), a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), and the like.

The storage unit 22 has a memory device that stores programs executed by the processor of the control unit 21 and various information, and is used as a working memory of the control unit 21 . The storage unit 22 includes a primary storage device (eg, RAM (Random Access Memory), ROM (Read Only Memory)) as a memory device. The storage unit 22 may include a secondary storage device (eg, HDD (Hard Disk Drive), SSD (Solid State Drive)) or a tertiary storage device (eg, optical disk, SD card) as a storage device. The storage unit 22 stores facility information 25 as management information regarding production facilities.

The communication unit 23 can communicate with another device such as the communication unit 33 of the terminal device 30 . Depending on the system configuration, the communication method by the communication unit 23 may use either wireless communication or wired communication between nodes. Various communication methods such as infrared communication, short-range wireless communication (for example, Bluetooth (registered trademark) communication), and mobile communication (for example, LTE (registered trademark)) may be included.

The terminal device 30 is configured by a worker computer, a working terminal, etc. arranged in a production line of a production facility, etc., and realizes each function. The terminal device 30 may include a tablet computer, a smart phone, a mobile communication terminal, and the like.

The terminal device 30 is connected to the management device 20 via a wired or wireless network 40 . The terminal device 30 has a control section 31 , a storage section 32 , a communication section 33 and a display section 34 . The control unit 31 has a processor, and realizes various functions by executing a predetermined program held in the storage unit 32 by the processor, for example. Processors may include MPUs, CPUs, DSPs, GPUs, and the like.

The storage unit 32 has a memory device that stores programs executed by the processor of the control unit 31 and various information, and is used as a working memory of the control unit 31 . The storage unit 32 includes a primary storage device (for example, RAM, ROM) as a memory device. The storage unit 32 may include a secondary storage device (eg, HDD, SSD) or a tertiary storage device (eg, optical disc, SD card) as a storage device.

The communication unit 33 can communicate with other devices such as the communication unit 23 of the management device 20, the imaging device 35, and the production equipment 50. FIG. Depending on the system configuration, the communication method by the communication unit 33 may use either wireless communication or wired communication between nodes. (for example, Bluetooth (registered trademark) communication), mobile communication (for example, LTE (registered trademark)), and other various communication methods may be included.

The display unit 34 displays various information display screens and various operation screens regarding the information management system 10 . The display section 34 may include a liquid crystal display device, an organic EL device, or other display devices. In addition, the display unit 34 displays a display screen such as a photographed image photographed by the photographing device 35 and a comparison result of confirmation processing using the photographed image. The display unit 34 may have a touch panel for operating the display screen.

In addition, the terminal device 30 is connected to the photographing device 35 . The photographing device 35 includes a light-emitting (illumination) unit that illuminates an object to be photographed, a camera, a scanner, and the like having an imaging device. The imaging device includes a solid-state imaging device such as a CCD (Charge Coupled Device) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor. Note that the photographing device 35 may include a hood portion that covers the camera and the light emitting portion. By bringing the hood portion into close contact with the subject during photographing, the photographed object can be reliably captured, and the influence of external light can be reduced.

Also, the terminal device 30 is connected to the production facility 50 . The production facility 50 includes various facilities related to production. The production facility 50 may have, for example, a molding machine that molds a product, and may include a controller for operating the molding machine, a sequencer (PLC (Programmable Logic Controller)), and the like. A molding machine of the production facility 50 is equipped with a mold 55 corresponding to a predetermined production process, and molding is performed by the mold 55 . Further, the terminal device 30 may include a voice input unit, and may accept operations by voice instead of the touch panel.

FIG. 2 is a diagram showing an example of equipment information in the embodiment. The equipment information 25 is various kinds of management information related to production equipment, and includes, for example, product information, production condition information, mold management information, and the like. The product information is information related to the product to be produced, such as product type, product name, product number, and product size. The production condition information is information relating to production steps (processes) corresponding to product information, and includes parameters that define conditions for each process. A parameter of the production condition information may include, for example, about 20 elements. The mold management information is information about the mold to be used corresponding to the product information and production condition information. The mold management information includes identification information such as letters, numbers, and symbols that can identify a predetermined mold, such as a management number and mold type.

Next, functions of the information management system 10 of this embodiment will be described. Here, at the site such as the production line of the production equipment, as inspection work of the production process, a confirmation process when a worker, a supervisor, or a work manager checks the correctness of a mold used in a predetermined production process I will explain the function of In the following description, a case is exemplified in which an operator becomes a user of the terminal device 30 and confirms by himself/herself the mold to be mounted.

Under the control of the control unit 21 , the management device 20 reads the facility information 25 from the storage unit 22 and transmits the facility information 25 to the terminal device 30 from the communication unit 23 . The terminal device 30 receives the facility information 25 from the management device 20 through the communication section 33 . When using the mold in a predetermined production process, the operator takes an image of the mold 55 to be confirmed by the photographing device 35 . The imaging device 35 transmits the captured image of the mold 55 to the terminal device 30 .

The terminal device 30 receives the photographed image of the mold 55 from the photographing device 35 through the communication unit 33 . The control unit 31 of the terminal device 30 performs image recognition processing on the photographed image of the mold 55 acquired from the photographing device 35, and identifies the mold management information of the mold 55 to be confirmed. As the image recognition processing of the photographed image, for example, template matching processing that performs matching determination by pattern matching with a template image is used to execute matching processing that determines the degree of matching with a reference master image. The control unit 31 of the terminal device 30 compares the mold management information included in the equipment information 25 acquired from the management device 20 and the mold management information obtained by image recognition of the captured image, and confirms that they match. do. When the mold management information of the facility information 25 and the mold management information of the photographed image match, the control unit 31 determines that the mold is proper, and outputs a determination result of successful matching (matching OK). The control unit 31 outputs a determination result of collation failure (collation NG) when the image recognition of the photographed image cannot be performed normally, or when the mold management information obtained by image recognition does not match.

Here, the control unit 31 includes a captured image acquisition unit 311 that acquires a captured image, and a master image acquisition unit 312 that acquires a master image. Note that other components such as the communication unit 33 may have the functions of the captured image acquisition unit and the master image acquisition unit.

A master image used as a template for image recognition of the photographed image of the mold 55 may be obtained from the management device 20 or may be stored in advance in the storage unit 32 or the like of the terminal device 30 . When acquiring from the management device 20 , the master image is pre-stored in the storage unit 22 of the management device 20 and read from the storage unit 22 . In the terminal device 30 , the master image sent from the management device 20 may be received by the communication section 33 and acquired by the control section 31 . When the master image is stored in advance in the terminal device 30 , the master image stored in the storage unit 32 may be read out and acquired by the control unit 31 .

When the determination result is that the photographed image is collated OK, the control unit 31 of the terminal device 30 outputs inspection completion information indicating completion of the confirmation process of the inspection work, and transmits the inspection completion information from the communication unit 33 to the production equipment 50. . The inspection work of the production process may include processing related to other inspection work other than confirmation processing of the photographed image of the mold. In the production facility 50, when the PLC of the molding machine receives the inspection completion information, it activates the sequence control for the molding machine to make the molding process ready to start. A molding machine is operable according to a predetermined sequence control by a PLC, and performs molding of a product.

Here, a problem in confirming the photographed image of the mold will be described. In the production process, in order to prevent mistakes in installing and selecting the molds to be worked on, we refer to the equipment information, and manage the mold management information for the specified work objects and the mold management information for the molds that are actually on hand. You need to make sure that the is consistent with As a method of confirming the mold management information, in general, the worker visually checks the mold management information of the mold at hand, confirms the value by voice or by manual input, and confirms the match of the mold management information on the terminal device. Judging is done. However, such a method is troublesome for the operator, and there is a possibility that an operation error such as an error in inputting a value may occur. For this reason, it is desirable to photograph the mold management information of the mold to be worked on and use confirmation processing based on image recognition of the photographed image to facilitate and ensure confirmation work including inspection of the mold.

In the confirmation process using the photographed image, even if the correct mold is prepared, due to the difference in the shooting angle when photographing the mold, the brightness of the subject, the light intensity of the lighting, and the irradiation angle of the light, etc. In some cases, it may not be possible to appropriately determine whether a master image used as a template and a photographed image to be checked match. In particular, in the case of metal molds, the illumination light makes it possible to determine the identification information in the photographed image. appears in the captured image. Since the surface of the mold is often formed in a curved shape, the light reflection state is not stable, and the collation result of the photographed image changes due to the influence of the reflection of the illumination light. In this case, the light condition of the subject changes every time the photograph is taken, and the light pattern in the photographed image differs, and the correct judgment result may not be obtained in the coincidence judgment with the master image. In addition, aging of the mold causes scratches, discoloration, corrosion (rust), and the like, and there is a risk of erroneous recognition occurring in matching determination between the photographed image and the master image.

FIG. 3 is a diagram showing examples of photographed images in a plurality of different modes when the mold is photographed. The photographed image of FIG. 3 is an example of an image photographed multiple times of the same portion of the surface of the mold on which the mold management information is stamped. The photographed image A is an image photographed from the left side of the mold as viewed in the photographing direction: the left side. The photographed image B is an image photographed from the right side of the mold in the photographing direction: the right side. The photographed image C is an image photographed from the front facing the mold in the photographing direction: front. As in these photographed images A to C, the mode of light differs depending on the photographing direction, and depending on the state of the reflected light, the die management information may not be properly read in the photographed images. A photographed image D is an image obtained by photographing a state in which scratches are added to the surface of the mold due to aging. If there is a flaw like this photographed image D, the die management information may be erroneously detected.

In the present embodiment, in view of the above circumstances, a configuration example of an information management apparatus capable of executing a simple and proper collation process will be shown. By performing image recognition using multiple master images, the recognition rate when matching photographed images to be confirmed can be improved. To enable stable collation in response to changes and the like. A plurality of master images may be used, for example, by using different reflection patterns of illumination light. In order to further improve the recognition rate, image recognition may be performed using a plurality of photographed images obtained by photographing a plurality of times, thereby improving the recognition rate when matching with the master image.

Note that the present disclosure is not limited to items to be checked that are made of metal or molds, and can also be applied to items that are made of materials or shapes that do not reflect light such as illumination light in a stable manner during photography. is.

Some specific examples of confirmation processing using captured images by the information management apparatus according to the present embodiment will be described below. The operation example of each embodiment shows a case where the control unit 31 of the terminal device 30 proactively executes processing.

The control unit 31 of the terminal device 30 performs image pattern matching by template matching processing, and compares the photographed image of the mold 55 to be confirmed with the master image serving as a template. At this time, the control unit 31 checks the photographed images of the mold using a plurality of (for example, ten) master images.

(Embodiment 1)
4 is a flowchart illustrating an operation example of confirmation processing using a photographed image by the information management apparatus according to Embodiment 1. FIG. Embodiment 1 shows a first example of a basic form relating to matching processing of photographed images. The photographed image is captured by the photographing device 35 and acquired by the photographed image acquisition section 311 of the control section 31 via the communication section 33 of the terminal device 30 . The master image is transmitted from the management device 20 , received by the communication unit 33 of the terminal device 30 , and acquired by the master image acquisition unit 312 of the control unit 31 .

The control unit 31 of the terminal device 30 first initializes the recognition conditions (S11). As a specific example of initialization, a counter for collating a plurality of master images in order is initialized to N=1. Then, the control unit 31 collates the N-th (first in the initial state) master image with the captured image (S12). In matching the master image and the captured image, the control unit 31 calculates a matching score indicating the degree of matching between the two images from the image recognition result, and determines whether the matching score is equal to or greater than a predetermined first threshold. It is determined whether the image and the master image can be regarded as matching (S13). The matching score indicates the error between the captured image to be matched and the master image. By comparing the photographed images, it is determined whether or not the mold management information in the photographed image matches the master image. It should be noted that pattern matching and OCR (Optical Character Recognition) may be linked in the matching process of the photographed images.

In step S13, if the matching score of the image matching is equal to or higher than the predetermined first threshold, the control unit 31 returns matching OK as the matching result of the photographed image of the mold (S14), and the matching OK is displayed on the display unit 34. output and display the matching result display. Then, the control unit 31 updates the N-th master image based on the collation result of collation OK (S15). At this time, the control unit 31 sets the master image as an update candidate so that the photographed image for which verification is OK can be replaced with the N-th master image. Note that the control unit 31 may actually perform the update processing of the master image automatically, or display a judgment screen on the display unit 34 or the like to allow the user to select whether or not to perform the update processing. good too. Then, the control unit 31 ends the confirmation process.

On the other hand, in step S13, if the matching score for image collation is less than the predetermined first threshold value, the control unit 31 determines whether the count value N of the current master image counter is greater than or equal to the number of master images ( S16). That is, if there are 10 master images, N=10, and it is determined whether all 10 master images have been collated. If the master image count value N is less than the number of master images (here, N<10), the controller 31 increments the master image count value N by 1 (N=N+1) (S17), and proceeds to step S12. Then, the N-th master image and the photographed image are collated. Thereafter, when the matching score of image collation is less than the predetermined first threshold, the processing of steps S12 to S17 is repeated. If the number of master images is 10, pattern matching between the photographed image and the master image is performed up to 10 times until collation is OK.

In step S16, if the count value N of master images is greater than or equal to the number of master images, that is, if the matching score is less than a predetermined first threshold in matching all the master images, the control unit 31 As a matching result of the photographed image of the mold, a matching NG is returned (S18), and the matching result display of the matching NG is output to the display unit 34 and displayed. In this case, it is conceivable that the wrong mold with a different mold control number was photographed as a confirmation target, or that the mold control number could not be photographed normally even if the mold was correct.

In step S18, the operator proceeds with the work even if it is actually determined that the collation is NG. For example, the worker performs work such as visually reconfirming the mold. In step S18, if the mold is a proper mold, the photographed image is added as a new master image, or the photographed image is replaced with any master image (that is, the master image is updated). , a process of storing the captured image may be performed. In this example, the master image is updated when the collation is OK, and the master image is added when the collation is NG. Therefore, the control unit 31 may allow the user to select update/addition for the new captured image. Then, the control unit 31 ends the confirmation process.

Here, a specific example of matching of photographed images by the control unit 31 will be described. For example, a photographed image of a mold is compared with 10 master images prepared in advance as a template by pattern matching to check whether the photographed image matches the master image. As a condition for determining that the mold management information of the mold in the photographed image is correct, a matching score with a maximum value of 100 in the case of a perfect match is used, and the matching score is 80 or more, which is the first threshold. In this case, the collation is OK. For example, if the matching score with the first master image (master 01) is 70 and the matching score with the second master image (master 02) is 85, it is determined that the matching is OK for the second master image. In this case, it may be possible to replace the second master image with the captured image captured this time.

When confirmation processing for the mold 55 to be confirmed is completed and a verification result indicating that the verification of the photographed image is OK is obtained, the terminal device 30 transmits inspection completion information for the mold 55 to the production equipment 50 . The worker attaches the confirmed proper mold 55 to the molding machine of the production facility 50 . Then, the control section of the production equipment 50 makes it possible to execute the next process, and the molding process is executed by a molding machine equipped with an appropriate mold 55 .

According to Embodiment 1, when performing confirmation using a photographed image to be confirmed, matching processing is performed using a plurality of master images, so that appropriate matching processing can be performed even when imaging conditions are different. It becomes possible. For example, when confirming an image by photographing a mold, etc., even if the amount of illumination light, the irradiation angle, the state of the reflected light, the photographing angle, etc., are different, it is possible to perform a matching judgment appropriately. It is possible to execute a simple and appropriate collation process.

(Embodiment 2)
FIG. 5 is a flowchart showing an operation example of confirmation processing using a photographed image by the information management device according to the second embodiment. Embodiment 2 shows a second example of a basic form relating to matching of photographed images.

In the second embodiment of the information management apparatus, the processes of steps S21 to S27 are the same as steps S11 to S17 of the first embodiment shown in FIG. That is, the control unit 31 of the terminal device 30 initializes the recognition conditions (N=1) (S21), compares the N-th master image and the captured image (S22), and the matching score reaches a predetermined value. If it is equal to or greater than the first threshold value (S23: Yes), a verification OK is returned as a verification result of the photographed image of the mold (S24), and the N-th master image is updated by the verification OK photographed image (S25). . Further, when the matching score is less than the predetermined first threshold (S23: No), the control unit 31 sequentially selects the N-th master image and the photographed image until the matching score becomes equal to or greater than the predetermined first threshold. is repeated until the master image count value N becomes equal to or greater than the number of master images (S26, S27, S22, S23).

In step S26, if the count value N of the master images is greater than or equal to the number of master images, that is, if the matching score is less than a predetermined first threshold in matching all the master images, the control unit 31 A re-photographing process by the device 35 is executed. Here, the control unit 31 determines whether or not the number of times of re-imaging is equal to or less than a predetermined number of times (for example, 3 times) (S28). Then, the user is prompted to retake the photographed image of the mold (S29). Then, when the matching score is less than the predetermined first threshold, the control unit 31 repeats the processing of steps S21 to S29, re-captures the captured image of the mold up to a predetermined number of times, and performs collation. If the number of master images is 10 and the predetermined number of times of re-capturing is 3, the pattern matching between the captured image and the master image is performed up to 10×3=30 times until collation is OK.

In step S28, if the number of times of re-imaging exceeds a predetermined number of times, that is, if the matching score is less than a predetermined first threshold value in collation of all master images even after performing re-imaging a predetermined number of times, the control unit 31 returns a verification NG as a verification result of the photographed image of the mold (S30), and outputs the verification result display of verification NG to the display unit 34 for display. In this case, it is conceivable that the wrong mold with a different mold control number was photographed as a confirmation object, or that even if it was a correct mold, the mold control number could not be photographed correctly after re-taking the prescribed number of times. . Then, the control unit 31 ends the confirmation process.

In the second embodiment, when a photographed image to be checked is collated using a plurality of master images, if collation fails, re-capturing is performed a predetermined number of times, and collation is performed using the re-captured photographed image. As a result, when confirming an image by photographing a metal mold or the like, even if the coincidence determination cannot be made correctly due to the difference in the photographing conditions, it is possible to carry out appropriate collation processing by re-photographing.

(Embodiment 3)
FIG. 6 is a flowchart showing an operation example of confirmation processing using a photographed image by the information management device according to the third embodiment. Embodiment 3 shows an example of processing for adding a master image.

In the third embodiment of the information management apparatus, the processes of steps S31 to S39 are the same as steps S21 to S29 of the second embodiment shown in FIG. That is, the control unit 31 of the terminal device 30 initializes the recognition conditions (N=1) (S31), compares the N-th master image and the photographed image (S32), and the matching score reaches a predetermined value. If it is equal to or greater than the first threshold value (S33: Yes), a verification OK is returned as a verification result of the photographed image of the mold (S34), and the N-th master image is updated by the verification OK photographed image (S35). . In addition, when the matching score is less than the predetermined first threshold (S33: No), the control unit 31 sequentially selects the N-th master image and the photographed image until the matching score becomes equal to or greater than the predetermined first threshold. is repeated until the master image count value N becomes equal to or greater than the number of master images (S36, S37, S32, S33). In addition, the control unit 31 executes re-imaging of the photographed image of the mold and repeats matching processing between the photographed image and the master image until the number of re-imaging exceeds a predetermined number of times (S38, S39, S31 to S37). . As in the first embodiment, re-capture may not be performed when the collation result of the photographed image is NG.

In step S38, if the number of times of re-imaging exceeds a predetermined number of times, that is, if the matching score is less than a predetermined first threshold value in collation of all master images even after performing re-imaging a predetermined number of times, the control unit 31 performs additional processing on the master image. In this case, it is conceivable that the wrong mold with a different mold management number was photographed as a confirmation target, or that even if it was a correct mold, a photographed image that matched the master image could not be obtained by re-taking the prescribed number of times. be done. Here, the control unit 31 displays the photographed images of the mold photographed multiple times on the display unit 34 and presents them to a user such as an operator so that the user can select the photographed image as a new master image. (S40). Then, the control unit 31 becomes ready to accept the selection information of the captured image.

FIG. 7 is a diagram showing an example of a captured image selection screen as a first example of a display screen by the information management device. The photographed image selection screen is an example of a screen displaying photographed images as addition candidates to be presented to the user for adding a master image.

On the photographed image selection screen 210, a first photographed image (image 1) 211, a second photographed image (image 2) 212, and a third photographed image (image 3) of the mold 55 to be confirmed are displayed. ) 213 are displayed side by side. An add button 214 and a cancel button 215 are arranged below the captured images 211 to 213 . When the user selects a desired photographed image by tapping, clicking, or the like and operates the add button 214 , selection information of a photographed image to be added as a master image is input to the control unit 31 . A plurality of images can be selected as the captured image to be added as the master image. When the user operates the cancel button 215, the process of selecting the captured image is cancelled.

The captured images to be added to the master image to be presented to the user may be presented by displaying all captured images that have been re-captured on the captured image selection screen 210, or by displaying a predetermined second image having a relatively high matching score. Captured images above the threshold may be extracted, displayed, and presented to the user. The second threshold may be any value smaller than the first threshold (first threshold>second threshold). More specifically, the second threshold may be, for example, a value slightly smaller than the first threshold (eg, 70). On the other hand, it is conceivable that an image that is completely different depending on the lighting conditions should be used as the master image even though the mold is appropriate. In this case, the second threshold may be 0, which indicates that the images are completely different, or a small value near 0.

When one or more captured images are selected in a state in which captured images are presented to the user on the captured image selection screen 210, and selection information is received (S41: Yes), the control unit 31 selects the captured images selected as master candidates. The image is added to the master image, and matching OK is returned as the matching result of the photographed image of the mold (S42). At this time, the control unit 31 outputs a master image addition completion display to the display unit 34 and causes the display unit 34 to display it. The master image addition completion display may include a display of the result of verification OK with the added master image. Then, the control unit 31 ends the confirmation process.

On the other hand, when the cancel button 215 is operated on the photographed image selection screen 210 and no photographed image is selected (S41: No), the control unit 31 returns verification NG as a verification result of the photographed image of the mold (S43). ), and outputs a collation result display of collation NG to the display unit 34 for display. Then, the control unit 31 ends the confirmation process.

The process for adding the master image is operated by the operator or the manager of the production process and executed by the control unit 31 of the terminal device 30. In addition, the manager of the information management system 10 operates the process to be executed by the manager of the management device 20. It may be configured to be executed in the control unit 21 . Also, it may be configured to be executed in another management terminal connected via the network 40 .

In the third embodiment, when a photographed image to be checked is collated using a plurality of master images, if the collation fails, the photographed image is presented to the user as an additional candidate for the master image, and the correct photographed image is determined by the user. A photographed image can be added as a new master image. As a result, even if matching cannot be correctly determined due to differences in photographing conditions when photographing a mold or the like and confirming the image, the added master image enables proper collation processing.

(Embodiment 4)
FIG. 8 is a flowchart showing an operation example of confirmation processing using a photographed image by the information management device according to the fourth embodiment. Embodiment 4 shows a first example of processing for updating a master image. The first example of master image update shows an example of update processing when collation is OK.

In the fourth embodiment of the information management apparatus, the processes of steps S51 to S59 are the same as steps S21 to S29 of the second embodiment shown in FIG. That is, the control unit 31 of the terminal device 30 initializes the recognition conditions (N=1) (S51), compares the N-th master image and the photographed image (S52), and the matching score reaches a predetermined value. If the value is equal to or greater than the first threshold (S53: Yes), a verification OK is returned as a verification result of the photographed image of the mold (S54), and the N-th master image is updated using the verification OK photographed image (S55). . Further, when the matching score is less than the predetermined first threshold (S53: No), the control unit 31 sequentially selects the N-th master image and the photographed image until the matching score becomes equal to or greater than the predetermined first threshold. is repeated until the master image count value N becomes equal to or greater than the number of master images (S56, S57, S52, S53). In addition, the control unit 31 re-captures the captured image of the mold and repeats the matching process between the captured image and the master image until the number of times of re-capturing exceeds a predetermined number (S58, S59, S51 to S57). . When the number of times of re-imaging exceeds a predetermined number of times, that is, when the matching score of all the master images is less than a predetermined first threshold even after re-imaging is performed a predetermined number of times (S58: No), and return NG as a result of matching the photographed image of the mold (S60). As in the first embodiment, re-capture may not be performed when the collation result of the photographed image is NG.

When updating the N-th master image in step S55, the control unit 31 compares the N-th master image that has been successfully collated, and compares the N-th master image before updating with the master image after updating. A comparison process with a photographed image is performed (S61). In the comparison process, a difference is calculated in a predetermined unit such as each block or pixel between the two images, and the amount of change in the captured image is obtained. Then, the control unit 31 determines whether or not there is a change in the photographed image for which the collation is successful, based on the comparison result (S62). Here, the control unit 31 recognizes that the amount of change in the photographed image is greater than or equal to a predetermined amount, and that scratches, discoloration, corrosion, etc., that are not affected by the reflection of light have occurred in the image, and that the image has changed. If so, the master image update process is performed for the master images other than the N-th master image (S63). When updating other master images, based on the amount of change in the captured image, similar to the N-th master image after updating, images of scratches, discoloration, etc. similar to the captured image are added to the other master images. Changes are added by image processing. Note that one or a plurality of images after image processing to which scratches, discoloration, etc. have been added may be presented to the user (operator or manager) as update candidates, and the master image may be updated according to the user's selection instruction. . Then, the control unit 31 ends the confirmation process.

In the fourth embodiment, when a photographed image to be checked is collated using a plurality of master images, even if the collation is successful, if an image change such as a scratch due to aging or the like is determined, another master image is used. A similar image change is also added by image processing. As a result, when photographing a mold or the like and confirming the image, even if the image changes due to aging or the like, it is possible to carry out appropriate matching processing.

(Embodiment 5)
FIG. 9 is a flowchart showing an operation example of confirmation processing using a photographed image by the information management device according to the fifth embodiment. Embodiment 5 shows a second example of processing relating to updating of the master image. The second example of master image update shows an example of update processing when collation is NG.

In the fifth embodiment of the information management apparatus, the processing of steps S71 to S79 is the same as steps S21 to S29 of the second embodiment shown in FIG. That is, the control unit 31 of the terminal device 30 initializes the recognition conditions (N=1) (S71), compares the N-th master image and the captured image (S72), and the matching score reaches a predetermined value. If it is equal to or greater than the first threshold (S73: Yes), a matching OK is returned as a matching result of the photographed image of the mold (S74), and the N-th master image is updated (S75). Further, when the matching score is less than the predetermined first threshold (S73: No), the control unit 31 sequentially selects the N-th master image and the photographed image until the matching score becomes equal to or greater than the predetermined first threshold. is repeated until the master image count value N becomes equal to or greater than the number of master images (S76, S77, S72, S73). Further, when the count value N of the master images is greater than or equal to the number of master images (S76: Yes), the control unit 31 re-captures the captured images of the mold until a predetermined number of times (for example, three times). (S78, S79, S71-S73, S76, S77). As in the first embodiment, re-capture may not be performed when the collation result of the photographed image is NG.

In step S78, if the number of times of re-imaging exceeds the predetermined number of times and all the master images are not verified even after the predetermined number of times of re-imaging, the control unit 31 compares the captured image with the predetermined master image to be compared. Perform comparison processing with the image. For example, even if the matching is NG, a comparison process is performed with a master image whose matching score is higher than a predetermined value (S80). For the master image to be compared, for example, a master image having a matching score higher than a predetermined second threshold is extracted and used. Note that a plurality of master images may be presented to the user (operator or manager), and a master image to be compared may be extracted according to the user's selection instruction. In the comparison process, a difference is calculated in a predetermined unit such as each block or pixel between the two images, and the amount of change in the captured image with respect to the master image is obtained. Then, the control unit 31 determines whether or not there is a change in the captured image based on the comparison result (S81).

Here, the control unit 31 recognizes that the amount of change in the photographed image is greater than or equal to a predetermined amount, and that scratches, discoloration, corrosion, etc., that are not affected by the reflection of light have occurred in the image, and that the image has changed. If so, the master image update process is performed for all the master images (S82). When updating the master image, the master image to be compared is updated with the photographed image, and based on the amount of change in the photographed image, similar to the master image to be compared, another master image is updated with the photographed image. Similar image changes such as scratches and discoloration are added by image processing. Note that the control unit 31 may present one or a plurality of images after image processing to which scratches, discoloration, etc. have been added as update candidates to the user, and may update the master image according to the user's selection instruction. Also, the currently captured image may be added as a new master image. After updating all the master images in step S82, the control unit 31 returns collation OK as a collation result of the photographed images of the molds (S83), and ends the confirmation process.

On the other hand, if the control unit 31 determines in step S81 that there is no change in the image (S81: No), the control unit 31 returns verification NG as a verification result of the photographed image of the mold (S84), and ends this process. .

FIG. 10 is a diagram showing an example of a master image confirmation screen as a second example of the display screen by the information management device. The master image confirmation screen is an example of a screen displaying a master image after image processing to be presented to the user for updating the master image.

On the master image confirmation screen 220, the master image after the image processing to which the image change is added is displayed as an update candidate. Here, for the master image of the mold 55 to be confirmed, image processing is performed to add similar scratches, discoloration, etc., corresponding to the image change detected in the master image to be compared (having a high matching score) used as a reference. is displayed after the image processing. In the illustrated example, the first master image 1 (master 1) 221 after image processing, the second master image 2 (master 2) 222 after image processing, . . . , the Nth master image N after image processing. (Master N) 22N is displayed in a list in the form of being arranged side by side. An approval button 224 and a discard button 225 are arranged below the master images 221 to 22N after image processing. When the user designates a master image whose update is to be approved by tapping, clicking, or the like, and operates the approval button 224 , approval information of the image-processed master image for which update is to be approved is input to the control unit 31 . The control unit 31 receives the approval information of the master image, and updates the master image to the approved master image after the image processing. A plurality of images can be designated as the master image after image processing to be updated. When the user operates the discard button 225, update processing of the specified master image is canceled.

As for the master images after the image processing of the update candidates to be presented to the user, a plurality of master images that have been subjected to image processing corresponding to the image changes of the captured images that have been captured a plurality of times are displayed on the master image confirmation screen 220, and the user can can be selected by presenting it to In addition, a plurality of master images subjected to image processing in different modes may be displayed on the master image confirmation screen 220 in response to image changes in the master image to be compared, and presented to the user for selection.

In the fifth embodiment, when a photographed image to be checked is collated using a plurality of master images, if the collation fails and an image change such as a scratch due to aging or the like is determined, the image is added to all the master images. Similar image changes are added by processing. As a result, when photographing a mold or the like and confirming the image, even if the image changes due to aging or the like, it is possible to carry out appropriate matching processing.

As described above, in this embodiment, the terminal device 30 or the management device 20 in the information management system 10 realizes the information management device and information management method according to this embodiment. The terminal device 30 acquires the photographed image of the mold 55 to be confirmed photographed by the photographing device 35 by the photographed image acquisition unit 311 . Also, the master image acquisition unit 312 acquires a master image that serves as a template for matching with the captured image. For example, the communication unit 33 receives a master image that has been stored in the management device 20 or the like and read out in advance, and the master image acquisition unit 312 acquires the master image. Alternatively, a master image stored and held in advance in the storage unit 32 is read from the storage unit 32 and acquired by the master image acquisition unit 312 . The control unit 31 of the terminal device 30 executes matching processing of the photographed images to be checked by performing match determination between the photographed image and the master image using a plurality of master images for the photographed image. At this time, the control unit 31 performs collation processing for the photographed image to be confirmed for the number of master images, and if it is determined that collation has failed, the photographed image is added as a new master image, or , the captured image is stored in the storage unit 32 in order to update the master image with the captured image. As a result, even if matching cannot be correctly determined due to differences in photographing conditions when photographing a mold or the like and confirming the image, appropriate collation processing can be performed by adding or updating the master image.

In addition, in the present embodiment, when the control unit 31 determines that the matching process has failed, the control unit 31 obtains the captured image by re-capturing the photographing device 35 up to a predetermined number of times, and performs the matching process again. As a result, when confirming an image by photographing a metal mold or the like, even if the coincidence determination cannot be made correctly due to the difference in the photographing conditions, it is possible to carry out appropriate collation processing by re-photographing.

Further, in the present embodiment, when the control unit 31 determines that the matching process has failed, the control unit 31 presents the acquired captured image to the user so that the captured image can be selected as a new master image. is received, the selected photographed image is added to the master image, and the determination result of matching success is output. Here, if the matching score, which indicates the degree of matching between the captured image and the master image, is equal to or greater than a predetermined value, the corresponding captured image may be presented to the user as a new master image candidate. As a result, the added master image enables more accurate collation processing. In addition, it is possible to extract photographed images having a matching score that is relatively high, and automatically present images that are more likely among the photographed images as master image candidates. Therefore, it is possible to perform a simple and appropriate collation process.

In addition, in the present embodiment, the plurality of master images includes images due to differences in the reflection state of the illumination light by the photographing device 35 . The item to be confirmed has a material or shape that allows identification information in the photographed image to be determined by illumination light irradiation, and changes the collation result of the photographed image due to the influence of reflection of the illumination light on the surface. For example, a metal mold or the like is taken as a confirmation target, irradiated with illumination light, photographed, image recognition of the photographed image is performed, and identification information such as mold management information is determined. A mold or the like is a material that is greatly affected by the reflection of illumination light due to the gloss of the surface or the like. Further, the surface is formed with a curved surface or the like, and has a shape in which the state of light in the photographed image, that is, the state of reflection of illumination light, changes depending on the photographing direction. In photographed images of such articles, variations in photographing conditions become large, and matching results of the photographed images vary. Even for such an article to be checked, it is possible to perform highly accurate matching processing using a plurality of master images.

Further, in the present embodiment, the control unit 31 performs matching processing on the photographed image to be confirmed, then compares the photographed image with the master image to be compared, and determines that there is an image change from the comparison result. In this case, the master image to be compared is updated with the photographed image, and the other master image is updated by adding the same image change as the photographed image by image processing. As a result, when a mold or the like is imaged and image confirmation is performed, highly accurate collation processing can be performed even if the image changes due to aging or the like.

In addition, in the present embodiment, when it is determined that the matching is successful in the matching process, the control unit 31 updates the master image by replacing the successfully matched master image with the photographed image, and replaces the successfully matched master image with the master image to be compared. The photographed image is compared with the successfully matched master image as a master image, and if it is determined that there is an image change from the comparison result, the same image change as the photographed image is added to the other master image by image processing. to update. As a result, even if collation is successful, if an image change such as a flaw due to aging or the like is determined, the same image change can be added to another master image by image processing, and the master image can be updated. Therefore, it is possible to perform a simple and appropriate collation process.

In addition, in the present embodiment, when the control unit 31 determines that the matching process has failed, the control unit 31 compares the captured image with a predetermined master image to be compared, and determines from the comparison result that there is an image change. In this case, all the master images are updated by adding the same image changes as the photographed images by image processing. As a result, when verification fails and an image change such as a scratch due to aging or the like is determined, the same image change can be added to all the master images by image processing, and the master images can be updated. Therefore, it is possible to perform a simple and appropriate collation process.

In addition, in the present embodiment, the control unit 31 sets a master image having a matching score higher than a predetermined value, which indicates the degree of matching between the captured image and the master image, as the master image to be compared. As a result, a master image with a relatively high matching score can be extracted and used as a comparison target, image changes can be determined, and highly accurate image changes can be added to the master image to update the master image.

In the present embodiment, the control unit 31 presents the master image after the image processing to which the image change is added as an update candidate to the user, receives the approval information of the master image, and receives the approved master image after the image processing. update to Thereby, the master image after image processing can be presented to the user, and the master image can be updated based on the user's approval instruction.

Further, in the present embodiment, the image change includes at least one of damage, discoloration, and corrosion in the item to be checked. As a result, when scratches, discoloration, corrosion, or the like occur in the item to be checked due to aging, the master image can be updated by reflecting image changes due to these scratches and the like. Therefore, it is possible to perform a more accurate collation process according to changes in the item to be checked.

In the above description, for ease of explanation, the flowcharts for the case of collation OK and the case of collation NG have been separately explained. On the other hand, in terms of terminal operation at the site, processing is performed to return only OK or NG of the verification result, and later update/addition processing of the master image using newly photographed images collectively taken by the administrator. Alternatively, another master image may be updated based on a newly captured image.

Conversely, even if the operator can update or add a master image using a newly captured image each time, or can update another master image based on a newly captured image. good.

Various embodiments have been described above with reference to the drawings, but it goes without saying that the present disclosure is not limited to such examples. It is obvious that a person skilled in the art can conceive of various modifications, modifications, substitutions, additions, deletions, and equivalents within the scope of the claims. Naturally, it is understood that it belongs to the technical scope of the present disclosure. In addition, the constituent elements of the various embodiments described above may be combined arbitrarily without departing from the gist of the invention.

INDUSTRIAL APPLICABILITY The present disclosure has the effect of making it possible to perform simple and appropriate matching processing when performing confirmation using a photographed image to be confirmed. It is useful as an information management device, an information management method, etc. in an information management system that performs confirmation processing.

10 information management system 20 management device 21 control unit 22 storage unit 23 communication unit 25 equipment information 30 terminal device 31 control unit 32 storage unit 33 communication unit 34 display unit 35 photographing device 40 network 50 production equipment 55 mold

Claims (11)

a photographed image acquisition unit that acquires a photographed image of an article to be confirmed from a photographing device;
a master image acquisition unit that acquires a master image that serves as a template for matching with the captured image;
a control unit that performs matching processing of the captured image by performing match determination between the captured image and the master image using a plurality of master images for the captured image;
a storage unit that stores information,
The control unit
performing collation processing on the photographed image for the number of the plurality of master images;
(i) when it is determined that collation has failed, presenting the acquired plurality of photographed images to the user, accepting selection of the photographed image as one of new master images, and transferring the selected photographed image to the plurality of master images; Add it to the image, output the result of successful matching,
(ii) updating the plurality of master images by replacing the photographed image that has been successfully matched with the master image that has been determined to have been successfully matched among the plurality of master images, if it is determined that the matching is successful;
Information management device. The information management device according to claim 1,
The control unit
If it is determined that the matching process fails, re-shooting is performed by the imaging device up to a predetermined number of times to obtain a captured image, and the matching process is performed again.
Information management device. The information management device according to claim 1 ,
The item to be confirmed is a mold or a metal part,
Information management device. The information management device according to any one of claims 1 to 3,
The plurality of master images include those due to differences in reflection conditions of illumination light by the imaging device,
Information management device. The information management device according to claim 2,
The control unit
When the number of times of re-imaging exceeds a predetermined number of times and it is determined that the collation has failed in the collation process, each of the captured images captured a plurality of times is displayed on a display unit, and selection is accepted as an additional master image.
Information management device. The information management device according to claim 1 ,
The control unit
If it is determined that the matching has failed in the matching process, and if the photographed image is compared with a predetermined master image to be compared , and if it is determined that there is an image change due to aging from the comparison result, all updating the master image by adding an image change similar to that of the captured image by image processing;
Information management device. The information management device according to claim 6 ,
The control unit
A master image for which a matching score indicating a degree of matching between the captured image and the master image is higher than a predetermined value is set as the master image to be compared.
Information management device. The information management device according to claim 6 ,
The control unit
presenting the master image after the image processing to which the image change is added as an update candidate to the user;
Receiving approval information of the master image, and updating to the approved master image after image processing;
Information management device. The information management device according to claim 6 ,
The image change due to aging includes at least one of scratches, discoloration, and corrosion of the item to be confirmed.
Information management device. Acquire a photographed image of the item to be confirmed from the photographing device,
Acquiring a master image that serves as a template for matching with the photographed image,
executing matching processing of the captured image by performing match determination between the captured image and the master image using a plurality of master images for the captured image;
In the matching process,
performing collation processing on the photographed image for the number of the plurality of master images;
(i) when it is determined that collation has failed, presenting the acquired plurality of photographed images to the user, accepting selection of the photographed image as one of new master images, and transferring the selected photographed image to the plurality of master images; Add it to the image, output the result of successful matching,
(ii) updating the plurality of master images by replacing the photographed image that has been successfully matched with the master image that has been determined to have been successfully matched among the plurality of master images;
Information management method. Acquire a photographed image of the item to be confirmed from the photographing device,
Acquiring a master image that serves as a template for matching with the photographed image,
executing matching processing of the captured image by performing match determination between the captured image and the master image using a plurality of master images for the captured image;
In the matching process,
performing collation processing on the photographed image for the number of the plurality of master images;
(i) when it is determined that collation has failed, presenting the acquired plurality of photographed images to the user, accepting selection of the photographed image as one of new master images, and transferring the selected photographed image to the plurality of master images; Add to the image, output the result of matching success,
(ii) updating the plurality of master images by replacing the photographed image that has been successfully matched with the master image that has been determined to have been successfully matched among the plurality of master images, if it is determined that the matching is successful;
having a computer carry out the processing;
program.

Images