JPWO2020100522A1 - Mark detection systems, signal processing circuits, computer programs and methods - Google Patents

Abstract

PROBLEM TO BE SOLVED: To speed up a search for a mark. An imaging device (300) of a mark detection system (1000) has a field of view capable of partially photographing an article to which a mark M is attached. The storage device (220) stores the database (DB) acquired by the mark position learning operation. The database contains at least one image feature amount of each image that does not include the mark image, and each image among a plurality of images obtained by photographing different parts of the article in a plurality of times by an imaging device. The positional relationship data indicating the relationship between the represented position and the mark position is associated with each other. When the image does not contain the image of the mark, the signal processing device compares the image feature amount of the acquired image with the image feature amount of each image in the storage device, and if the comparison results match, the signal processing device is in the database. , The position of the imaging device is changed by using the positional relationship data associated with the matched image, and then the image of the mark is detected from the image taken again.

Description

The present disclosure relates to mark detection systems, signal processing circuits, computer programs and methods.

A mark detection system is known in which a work is photographed with a camera and an alignment mark (hereinafter, simply referred to as "mark") given to the work is detected by image processing. Determining the position and / or orientation of the work based on the position and orientation of the alignment marks allows for subsequent processing, such as gripping and unloading the work by a robot.

Japanese Unexamined Patent Publication No. 2012-138548 discloses a technique for detecting a mark by moving the field of view in order from the periphery of the imaging reference position when the mark cannot be detected at the imaging reference position.

Japanese Unexamined Patent Publication No. 2012-138548

By the way, in the above-mentioned conventional technique, it takes time to search for a mark.

The mark detection system of the present disclosure is a mark detection system that detects a mark by using an image pickup device in an exemplary embodiment, and is an image pickup device having a field of view capable of partially photographing an article to which the mark is given. , A storage device that stores a database acquired by the mark position learning operation, and the database is one of a plurality of images obtained by photographing different parts of the article in a plurality of times by the imaging device. For a plurality of images that do not include the image of the mark, at least one image feature amount of each image and positional relationship data indicating the relationship between the position represented by each image and the position of the mark are related. A storage device and a signal processing device are provided, and at the time of mark detection operation, the signal processing device causes the image pickup device to take a picture of a part of the article, and the image acquired from the image pickup device includes the image of the mark. If not, the image feature amount of the acquired image is compared with the image feature amount of each image stored in the storage device with reference to the database, and if the comparison results match, The positional relationship data associated with the matching image is read from the database, and the read positional relationship data is used to change the position of the imaging device so that the mark is placed in the field of view of the imaging device. After generating and outputting a control signal for this purpose and changing the position, the image of the mark is detected from the image taken again by the imaging device.

The signal processing circuit of the present disclosure is a signal processing circuit used in a mark detection system that detects using an image pickup apparatus in an exemplary embodiment, and the mark detection system partially displays a marked article. An image pickup device having a field that can be photographed and a storage device that stores a database acquired by a mark position learning operation. The database captures different parts of the article in a plurality of times by the image pickup device. Among the plurality of images obtained above, for a plurality of images that do not include the image of the mark, at least one image feature amount of each image and the relationship between the position represented by each image and the position of the mark are shown. The signal processing circuit includes a storage device associated with positional relationship data, and the signal processing circuit causes the image pickup device to take a picture of a part of the article at the time of mark detection operation, and an image acquired from the image pickup device. When the image of the mark is not included in, the image feature amount of the acquired image is compared with the image feature amount of each image stored in the storage device with reference to the database, and the comparison result is obtained. If they match, the positional relationship data associated with the matched image is read from the database, and the read positional relationship data is used so that the mark is placed in the field of view of the imaging device. A control signal for changing the position of the image pickup device is generated and output, and after the position is changed, the image of the mark is detected from the image taken again by the image pickup device.

The computer program of the present disclosure is a computer program for operating a signal processing circuit used in a mark detection system that detects a mark by using an image pickup apparatus in an exemplary embodiment, and the mark detection system is a mark. An image pickup device having a field of view capable of partially photographing an article to which the article is attached, and a storage device for storing a database acquired by a mark position learning operation, wherein the database captures different parts of the article with the image pickup device. Of the plurality of images obtained by taking the images in a plurality of times, for a plurality of images that do not include the image of the mark, at least one image feature amount of each image, a position represented by each image, and the above. It is provided with a storage device that associates positional relationship data indicating a relationship with the position of the mark, and during the mark detection operation, the computer program controls the image pickup device to the signal processing circuit. When a part of the article is photographed and the image acquired from the imaging device does not include the image of the mark, the image feature amount of the acquired image and the storage device are stored with reference to the database. The image features of each of the above images are compared, and if the comparison results match, the positional relationship data associated with the matching image is read out from the database, and the read out positional relationship data is used. Then, a control signal for changing the position of the image pickup device is generated and output so that the mark is placed in the field of view of the image pickup device, and after the position is changed, an image is taken again using the image pickup device. The image of the mark is detected from the image.

In an exemplary embodiment, the method of the present disclosure is a method performed using a mark detection system that detects a mark using an image pickup apparatus, and the mark detection system partially covers an article to which the mark is attached. An image pickup device having a field that can be photographed and a storage device that stores a database acquired by a mark position learning operation. The database captures different parts of the article in a plurality of times by the image pickup device. Among the plurality of images obtained in the above, for a plurality of images that do not include the image of the mark, at least one image feature amount of each image and the relationship between the position represented by each image and the position of the mark are determined. It is provided with a storage device and a signal processing device that associate the positional relationship data to be shown. It is determined whether or not the image acquired from the image pickup apparatus contains the image of the mark, and when the image acquired from the image pickup apparatus does not include the image of the mark, the signal processing circuit is provided with the database. With reference, the acquired image feature amount of the image is compared with the image feature amount of each image stored in the storage device, and if the comparison results match, the matching image is displayed in the database. Control for reading out the associated positional relationship data and changing the position of the imaging device so that the mark is placed in the field of view of the imaging device by using the positional relationship data in the signal processing circuit. After generating and outputting a signal and changing the position, the signal processing circuit is made to detect the image of the mark from the image taken again by the imaging device.

According to the embodiments of the present disclosure, a mark detection system, a signal processing circuit, a computer program and a method capable of searching for a mark in a short time are provided.

FIG. 1 is a diagram schematically showing the mark detection system 1000 of the present disclosure. FIG. 2 is a diagram showing a connection relationship between the articulated robot 100, the controller 200, and the imaging device 300, and a hardware configuration example of the controller 200. FIG. 3 is a diagram showing an example of the shapes of the marks M and M1 to M3 according to the present disclosure. FIG. 4 is a diagram showing a top plate 12 divided into 30 regions. FIG. 5 is a diagram showing images GA3 and GA4 of regions A3 and A4 captured by the image pickup apparatus 300. FIG. 6 is a diagram showing an example of a plurality of partial images in which the image GA3 is divided for pattern matching. FIG. 7 is a diagram for explaining the relationship between the position represented by the image GA3 and the position of the mark M. FIG. 8 is a diagram showing a process of extracting an image feature amount from a partial image 3-1. FIG. 9 is a diagram showing an example of a database DB in which data indicating a misalignment amount and an image feature amount (edge image file) are associated with each other. FIG. 10 is a diagram showing a region P1 first captured by the imaging device 300. FIG. 11 is a diagram showing a concept of processing for comparing the image feature amount of the partial region 3-3 with the image feature amount of each of the plurality of partial regions 252. FIG. 12 is a diagram showing an image GA3 of the region A3 used for learning and an image GP1 of the region P1 taken by the image pickup apparatus 300. FIG. 13 is a diagram showing a concept of processing for comparing the image feature amount of the partial region 2-3 included in the image GA3 with the image feature amount of each of the plurality of partial regions 252 set in the image GP1. .. FIG. 14 is a diagram showing the correspondence between each partial region of the image GA3 of the region A3 used for learning and the partial region of the image GP1 of the region P1 having the smallest sum of squared errors by arrows. be. FIG. 15 is a flowchart showing a procedure of processing executed by the CPU 210. FIG. 16 is a diagram showing an example in which the center C of the field of view of the image pickup apparatus 300 after movement and the center point O of the image GM of the mark do not match. FIG. 17 is a diagram showing new positional deviations Δx'and Δy' after updating the positional deviations Δx and Δy. FIG. 18 is a diagram showing an example of the updated database DB for image GA3. FIG. 19 is a diagram showing an example of a database DB for image GA3 in which priority is introduced.

Hereinafter, embodiments of the mark detection system according to the exemplary embodiment will be described with reference to the accompanying drawings. In the present specification, an unnecessarily detailed description may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. It should be noted that the inventor of the present invention provides the accompanying drawings and the following description in order for those skilled in the art to fully understand the present disclosure, and intends not to limit the subject matter described in the claims by these. In the following description, the same or similar components are designated by the same reference numerals.

<Termin> An "automated guided vehicle" (AGV) means an automated guided vehicle that manually or automatically loads luggage into the main body, automatically travels to a designated place, and manually or automatically unloads. "Automated guided vehicles" include automated guided vehicles and unmanned forklifts. The "automated guided vehicle" has a mounting table on which articles are placed. The mounting table can broadly include an object on which an article can be placed, such as a top plate and an arm.

An "object" is a work or an object that includes a platform for the work. A "work" is a variety of objects such as products or parts. In one example, when the article is a workpiece, the article is placed on the top plate of the AGV and transported, loaded onto the carrier by a manual or pickup robot, and carried out.

The "mark" is a mark that can be identified from others based on an image that can be acquired by an imaging device such as a camera. A "mark" may include letters, graphics, symbols or colors, or a combination thereof. Line diagrams, patterns, and three-dimensional shapes with specific patterns are also included in the "mark". The mark used for positioning is called an "alignment mark". When a "mark" such as a specific shape that is not provided assuming positioning is used for positioning, the "mark" is an "alignment mark". In the present disclosure, an example in which the "mark" is used for positioning will be described. In that case, "mark" is synonymous with "alignment mark".

Hereinafter, an exemplary mark detection system according to the present disclosure will be described. The mark detection system is used to detect a predetermined mark using an imaging device. In particular, the mark detection system according to the present embodiment performs a process capable of quickly moving the image pickup device to the position of the mark when the mark does not exist in the field of view of the image pickup device.

FIG. 1 schematically shows the mark detection system 1000 of the present disclosure. The mark detection system 1000 of the present disclosure includes an articulated robot 100, a controller 200, and an imaging device 300 in a non-limiting and exemplary embodiment. The mark detection system 1000 detects a mark given to the mounting table on which the work is placed, or a mark given to the work itself having a size that does not fit in the field of view of the image pickup apparatus. An example of a mounting table is an AGV top plate, while another example is a movable table or a fixed table.

In an exemplary embodiment, the AGV 10 carries the work W on the top plate 12 and stops at a predetermined position. The mark M is attached to the mounting table of the work W in advance. The mark detection system 1000 detects the mark M using the image pickup apparatus 300. Upon detecting the mark M, the mark detection system 1000 determines the position of the work, that is, the position and / or orientation of the work, based on the position and orientation of the mark M. After positioning, the mark detection system 1000 picks up the work W using the articulated robot 100 and sends it to a later process. The specific content of the process of detecting the mark M using the image pickup apparatus 300 will be described in detail later.

The articulated robot 100 and the controller 200 shown in FIG. 1 are fixed to the floor, and such an installation method is an example. In another example, the articulated robot 100 and the controller 200 may be miniaturized and installed on another AGV different from the AGV10.

In still another example, the work may be transported by a trolley pushed by an operator instead of being transported by the AGV10. When the operator pushes the dolly by hand, the position of the dolly does not always stop at the same position, but rather may be different each time. Even in such a case, if the mark detection system according to the present disclosure described later is used, the mark can be detected in a short time, the position and / or orientation of the work can be determined, and the work can be picked up.

Hereinafter, the articulated robot 100, the controller 200, and the imaging device 300 will be described.

The articulated robot 100 is also commonly referred to as a robot arm or manipulator. The articulated robot 100 has joints 110A to 110C, arms 120A and 120B, and an end effector 130. The joints 110A to 110C each have a motor that rotates the arms 120A and / or 120B around a rotation axis. The end effector 130 is a device attached to the arm 120B so that the articulated robot 100 can perform work. The exemplary end effector 130 shown in FIG. 1 has a two-finger gripper 140. The two-finger gripper 140 can adjust the distance between two fingers to sandwich and grip the work W.

An example of the controller 200 is a computer. In the present embodiment, the controller 200 causes the image pickup apparatus 300 to take a picture, acquires an image signal obtained by the picture, and performs a process of detecting a mark. When the mark is detected, the controller 200 performs a predetermined positioning process to determine the position and / or orientation of the work W. The controller 200 then controls the articulated robot 100 and the two-finger gripper 140 based on the determined position and / or orientation to pick up the work W. It is not essential that the controller 200 performs all the above-mentioned processes. For example, the control of the articulated robot 100 and the process of detecting the mark may be performed by separate computers.

The image pickup device 300 is a device that outputs an image signal of an image of a photographed subject. In this embodiment, the image signal is sent to the controller 200 by wire. A typical example of the image pickup apparatus 300 is an imaging apparatus including an area sensor such as a CMOS image sensor or a CCD image sensor in which a large number of photodiodes are arranged in a matrix. The image pickup apparatus 300 generates data of a color image or a monochrome image of a subject. As the image pickup apparatus 300, for example, various cameras for visual inspection can be used.

Now, as shown in FIG. 1, it is assumed that the work W is smaller than the top plate 12. The image pickup apparatus 300 has a field of view relatively narrower than that of the work W, in which the work W can be partially photographed. The term "field of view" as used herein means, for example, that the articulated robot 100 takes a predetermined posture and the imaging device 300 faces the subject to acquire features such as the shape of the mark M in order to photograph the mark M. The field of view when located at a possible distance.

The reason why the field of view of the image pickup apparatus 300 is narrow is as follows. The image pickup apparatus 300 used for detecting the mark and determining the position and / or orientation of the work is assumed to be used for high-precision work in other processes, and employs a high-magnification lens. ing. Therefore, the field of view is inevitably narrowed. In order to widen the field of view, it is conceivable to provide at least two imaging devices for mark detection and positioning and an imaging device for high-precision work, and adopt a low-magnification lens for the former. Alternatively, it is conceivable to adopt a lens whose magnification can be changed arbitrarily and set the magnification to a low value at the time of mark detection. However, the adoption of a two-unit configuration or a variable magnification lens becomes complicated and leads to an increase in cost. Further, since the weight of the device or the lens increases, mounting may not be permitted in the first place due to restrictions on the weight that can be mounted on the end effector.

FIG. 2 shows the connection relationship between the articulated robot 100, the controller 200, and the imaging device 300, and a hardware configuration example of the controller 200. The controller 200 has a CPU 210 which is a signal processing device and a memory 220 which is a storage device.

The controller 200 is connected to the articulated robot 100 and the imaging device 300 by wire via one or a plurality of communication terminals (not shown). The controller 200 and the articulated robot 100 transmit and receive signals via the communication terminal. Further, the controller 200 and the image pickup apparatus 300 transmit and receive a shooting instruction signal and an image signal acquired by shooting via the communication terminal. The communication may be performed wirelessly.

The signal transmitted from the controller 200 to the articulated robot 100 may include designation of one or more joints of the articulated robot 100, designation of a rotation angle for each designated joint, designation of a rotation speed, and the like. The signal transmitted from the articulated robot 100 to the controller 200 may include confirmation of reception of the transmitted signal and notification of completion of operation.

A computer program is stored in the memory 220. A computer program is a set of instructions executed by the CPU 210. When the CPU 210 executes the computer program of the controller 200, the CPU 210 executes a shooting process using the image pickup apparatus 300 and a mark M detection process using the acquired image. When it is determined that the image acquired from the image pickup apparatus 300 contains the image of the mark M, the CPU 210 executes a predetermined positioning process using the detected mark M. Based on the result of the positioning process, the CPU 210 can rotate each motor of each joint of the articulated robot 100 and pick up the work W.

In the present embodiment, the operation of the mark detection system 1000 is controlled by the CPU 210 of the controller 200, but if the CPU 210 appears in all the explanations, the description becomes complicated. Therefore, in the following, it may be simply described as "the image pickup apparatus 300 captures a predetermined area." The operation executed by this expression is that the CPU 210 actually rotates each joint of the articulated robot 100 to control the field of view of the imaging device 300 so as to enter a predetermined region, and the CPU 210 controls a shooting instruction signal (control signal). ) Is transmitted to the image pickup apparatus 300 to cause the image pickup apparatus 300 to acquire an image signal in the field of view, at least. Further, it may be included that the CPU 210 acquires an image signal from the image pickup apparatus 300.

The memory 220 stores a database DB in which various data after learning, which will be described later, are associated with each other. In the database DB, for a plurality of images prepared for learning that do not include a mark image, at least one image feature amount of each image and a position indicating the relationship between the position represented by each image and the mark position are shown. Relationship data, is associated. More specific contents of the database DB will be described later.

FIG. 3 shows an example of the shapes of the marks M and M1 to M3 according to the present disclosure. In this example, four marks are given near the center of each side of the top plate 12 of the AGV 10. Each mark has a different pattern from each other, and the detected position can be determined by the position and orientation of the detected pattern. In FIG. 3, a relatively large mark is shown for convenience of description. The mark may be smaller. The number, position, shape (pattern), etc. of the marks are arbitrary. One of ordinary skill in the art can increase or decrease the number and change the position and pattern as appropriate to the extent required.

The memory 220, which is a storage device, holds the above-mentioned data such as the number, position, and shape of marks. By using these data, the CPU 210 can identify the image of the mark from the image acquired from the image pickup apparatus 300. In the present specification, such data used for identifying the mark image from the captured image is referred to as "feature data" related to the mark image. The CPU 210 refers to the feature data and determines whether or not the captured image contains an image of the mark. Specifically, the CPU 210 refers to the feature data in the memory 220 to determine whether or not an image object having features matching the feature data exists in the captured image. If there is an image object having features that match the feature data, the CPU 210 determines that the image contains an image of the mark. If such an image object does not exist, the CPU 210 determines that the image does not contain an image of the mark. If the feature data is prepared in advance, the CPU 210 can detect an arbitrary mark.

Hereinafter, the mark detection operation performed by the mark detection system 1000 according to the present disclosure will be described. The mark detection operation performed by the mark detection system 1000 includes a mark position learning operation and a mark detection operation. The mark position learning operation is performed prior to the first mark detection operation.

The mark position learning operation is a learning operation performed by the mark detection system 1000 using a plurality of images taken for learning. The object of learning is the relationship between the amount of image features of an image that does not include a mark and the amount of misalignment from the position represented by each image to the center position of the mark (hereinafter referred to as "displacement amount"). In the present embodiment, at least a part of the top plate 12 of the AGV 10 is taken into the field of view by the image pickup apparatus 300, and the image is taken in a plurality of times. Of the plurality of images obtained as a result, for a plurality of images that do not include the mark image, at least one image feature amount of each image, for example, an image feature amount related to an edge in the image, and a mark image are included. Learn the relationship with the amount of deviation from the position represented by each image (for example, the center position of the field of view at the time of taking each image) to the center position of the mark. The positional relationship data indicating the image feature amount and the deviation amount of the image not including the mark, which is the learning result, is associated and stored as a database DB, and is used in the next mark detection operation.

The mark detection operation is an operation in which the CPU 210 of the mark detection system 1000 causes the image pickup device 300 to take an image and uses the image acquired from the image pickup device 300 to detect the mark. If the acquired image contains an image of the mark, the mark may be detected from the image as it is. On the other hand, when the acquired image does not include the image of the mark, the CPU 210 of the mark detection system 1000 refers to the database and refers to the image feature amount of the acquired image and the learned image feature amount stored in the database DB. Compare with. When the comparison result matches a predetermined condition, the CPU 210 determines that the two image feature amounts match. An example of a predetermined condition is whether or not the magnitude of agreement between the two is less than a predetermined threshold value. For example, consider the case where the position and range of an edge in an image are used as an image feature amount. The difference between the image feature amount of the acquired image and the image feature amount stored in the database DB is obtained, and the sum of squares of the difference is obtained. If the magnitude of the sum of squares is less than the threshold value, it can be said that the difference between the two image features is small. That is, it can be said that the acquired image includes the same area or an overlapping area as the image used for learning. The image feature amount may be obtained from the entire acquired image, or may be obtained from a partial image that is a part of the acquired image.

Therefore, the CPU 210 further refers to the database DB and reads out the positional relationship data indicating the amount of deviation to the center position of the mark associated with the image used for learning. When the image pickup apparatus 300 is moved by the amount of the deviation, a mark is formed in the field of view of the image pickup apparatus 300. Therefore, the CPU 210 uses the read positional relationship data to generate and output a control signal for changing the position of the image pickup device 300 so that the mark is placed in the field of view of the image pickup device 300. After changing the position of the image pickup device 300, the CPU 210 detects the image of the mark from the image taken again by the image pickup device 300. As a result, even if the acquired image does not include the image of the mark, the CPU 210 can move the image pickup apparatus 300 to the position of the mark and detect the mark. Since it is not necessary to adopt a method of detecting a mark by photographing the entire area while changing the photographing range (a brute force detection method), it is possible to quickly detect the position of the mark.

The mark position learning operation and the mark detection operation will be described below.

<Mark position learning operation> FIG. 4 shows the top plate 12 divided into 30 regions. For convenience of explanation, it is assumed that each region is equal to the size of the field of view of the image pickup apparatus 300. In FIG. 4, the numbers A1, A2, ..., A6 are assigned in order from the upper left area to the right, and the numbers B1, B2, ... Are assigned in order to the right in the next line. By the same rule, numbers are assigned up to the last area E6 in the fifth row (column E).

The image pickup apparatus 300 photographs each region by a brute force detection method. Pay attention to line A at the top. A mark M is given to the area A4. As a result of photographing by the image pickup apparatus 300, the image of the mark is included in the image of the area A4, and the image of the mark is not included in the image of the areas A1 to A3, A5 and A6. In addition to the image of the top plate 12, the image of the region in row A may include an image of the floor or the like below the top plate 12. Hereinafter, the regions A3 and A4 will be focused on and described.

FIG. 5 shows the images GA3 and GA4 of the regions A3 and A4 captured by the image pickup apparatus 300. In FIG. 5, the images GA3 and GA4 are arranged adjacent to each other for convenience of explanation. In the image GA4, there is a mark image GM which is an image of the mark M. On the other hand, although the image GA3 does not include the mark image GM, it is known that the mark image GM exists in the adjacent image GA4 at the time of learning. Therefore, in the present embodiment, the positional relationship data indicating the relationship between the position represented by the image GA3 and the position of the mark M is acquired.

The relationship between the position represented by the image GA3 and the position of the mark M is uniquely determined with reference to a given point (example: center point) in the image GA3.

FIG. 6 shows an example of a plurality of partial images in which the image GA3 is divided for pattern matching described later. In the example of FIG. 6, the image GA3 is divided into 16 partial images, each having the same shape and area.

FIG. 7 is a diagram for explaining the relationship between the position represented by the image GA3 and the position of the mark M. For convenience of explanation, the positions of the image GA3 and the image GM of the mark will be used for explanation. The position of the center point C of the image GA3 corresponds to the position of the center of the field of view of the image pickup apparatus 300. In the present embodiment, the CPU 210 of the controller 200 previously sets a deviation amount (Δx, Δy) indicating how much the image pickup apparatus 300 should be moved from the position where the image GA3 is acquired so that the center of the mark M coincides with the center of the field of view. Acquire and store it in the memory 220.

FIG. 7 shows the deviation amounts Δx and Δy for each coordinate axis from the center point C of the visual field when the image GA3 is taken to the center point O of the mark M. The illustrated Δx and Δy are Δx = + 30 mm and Δy = −7 mm, respectively. The X-axis symbol "+" indicates the right direction of the drawing, and the Y-axis symbol "-" indicates the upper part of the drawing. This value may be measured and input by the operator of the mark detection system 1000, or may be calculated and obtained by the CPU 210. In the present embodiment, the same amount of deviation (Δx and Δy) is associated with all the partial images 1-1, 1-2 and the like constituting the image GA3.

In this way, the deviation from the center position of the field of view at the time of shooting to the center position of the mark M for each image that does not include the image GM of the mark, for example, the images obtained by capturing the regions A1, A2, A3, etc. shown in FIG. The quantity (Δx, Δy) can be obtained. When a plurality of marks M exist, the amount of deviation Δxy up to each mark M is calculated, and the smallest of them can be adopted as data indicating the relationship between the position represented by the image and the position of the mark.

Next, the image feature amount related to the above-mentioned deviation amount will be described by exemplifying the partial image 3-1 of FIG.

FIG. 8 shows a process of extracting an image feature amount from a partial image 3-1. The CPU 210 of the controller 200 performs a process of extracting the edge included in the partial image 3-1 and adopts the data of the extracted edge image 3-1-E as the image feature amount of the partial image 3-1. The CPU 210 stores the image feature amount in the memory 220 as a database DB in relation to the data indicating the relationship (deviation amount) between the position represented by the image and the mark position described above. In the present embodiment, the edge in the partial image is adopted as the image feature amount, but this is only an example. Any feature unique to each partial image can be adopted as the image feature amount.

FIG. 9 shows an example of a database DB in which data indicating a deviation amount and an image feature amount (edge image file) are associated with each other. The CPU 210 creates a database DB for all the images obtained by capturing the area shown in FIG. 4 (excluding the images including the image GM of the mark), and stores the database DB in the memory 220. It is optional whether the database DB for each image is provided as a separate file or the database DB for each image is collectively included in one file.

By creating the database DB of each image by the above procedure, the mark position learning operation is completed.

<Mark detection operation> When the mark position learning operation is completed, the mark can be detected using the mark detection system 1000. At this point, the image feature amount is extracted from each image obtained by photographing each of the regions A1 to E6 shown in FIG. 4 and stored in the database DB.

The mark detection operation is executed after the AGV 10 conveys the work W and stops at a predetermined position in front of the mark detection system 1000 as shown in FIG. The image pickup apparatus 300 photographs the top plate 12 on which the work W is placed.

Now, as shown in FIG. 10, it is assumed that the image pickup apparatus 300 first photographs the region P1. The region P1 first captured by the image pickup apparatus 300 in the field of view does not necessarily include the center position of the top plate 12.

The CPU 210 of the controller 200 refers to the database DB and determines in which area of the learned areas the image feature amount included in the area P1 is included in the image feature amount. Specifically, the CPU 210 repeats the following processing from, for example, the area A1 (FIG. 4) to the final area E6 (FIG. 4).

First, the CPU 210 compares the image feature amount of each partial area included in the area A1 (FIG. 4) with the image feature amount of each of the plurality of partial areas set in the area P1. FIG. 11 shows the image feature amount of the partial region 3-3 included in the image GA1 of the region A1 used for learning, and the image feature amount of each of the plurality of partial regions 252 set in the image GP1 of the region P1. Shows the concept of the process of comparing. The plurality of partial regions 252 can be set, for example, while shifting by several pixels and overlapping with other partial regions. In this embodiment, the size of the partial region 3-3 and the size of each of the plurality of partial regions 252 are the same. This operation can be performed using known template matching.

For example, the CPU 210 obtains the sum of the image feature amount of the partial area 3-3 and the square error of each image feature amount of the plurality of partial areas 252. The magnitude of the sum of the squared errors obtained is a parameter for evaluating how close the image in each partial region 252 is to the image in the partial region 3-3. The smaller the sum of the squared errors, the more similar the images in the two subregions.

However, as can be understood from the fact that the two images GA1 and GP1 shown in FIG. 11 do not match at first glance, the degree to which each partial region in the image GA1 coincides in the image GP1 is low. The CPU 210 determines that the number of sets of subregions in which the magnitude of the sum of squared errors is equal to or less than the predetermined value T1 does not exist between the two images, which is also the predetermined value T2 or more. In such a case, it may be expressed in the present specification that the comparison results of the two images do not match. On the contrary, when the number of pairs of subregions in which the magnitude of the sum of the square errors is equal to or less than the predetermined value T1 exists between the two images, the number of pairs of the subregions is also equal to or greater than the predetermined value T2. It may be expressed that the comparison results of the images match.

The CPU 210 calculates the sum of the squared errors of each partial region of each image of the regions A2 to E6 used for learning and each partial region in the captured image GP1. In the following, as an example in which the comparison results of the two images match, the processing related to the image in the region A3 (FIG. 4) used for learning and the image GP1 will be illustrated.

FIG. 12 shows the image GA3 of the region A3 used for learning and the image GP1 of the region P1 taken by the image pickup apparatus 300. Since the shooting environment is different between the two, the obtained images are also different. FIG. 13 shows the concept of a process of comparing the image feature amount of the partial region 2-3 included in the image GA3 with the image feature amount of each of the plurality of partial regions 252 set in the image GP1. Similar to the above example, the CPU 210 calculates the sum of the squared errors between the image feature amounts for each partial region set in the image GP1.

FIG. 14 shows by arrows the correspondence between each partial region of the image GA3 of the region A3 used for learning and the partial region of the image GP1 of the region P1 having the smallest sum of squared errors. .. When the number of sets of subregions in which the magnitude of the sum of the square errors is equal to or less than the predetermined value T1 between the two images exists in the same predetermined value T2 or more, the CPU 210 determines the region. It is determined that the image GA3 of A3 and the image GP1 of the area P1 match. Since the threshold values T1 and T2 can vary greatly depending on the size of the images GA3 and GP1, the number of subregions to be set, and the like, the listing of specific examples is omitted. A person skilled in the art can appropriately set it according to the conditions.

When the comparison results of the two images GA3 and GP1 match by the above processing, the CPU 210 refers to the database DB and acquires the positional deviations Δx and Δy. Then, the CPU 210 moves the image pickup apparatus 300 by the positional deviation Δx and Δy. As a result, the image pickup apparatus 300 can be quickly and surely moved to the position of the mark even from the image obtained by capturing the area not including the mark.

Although FIG. 12 gives an example in which the image GA3 and the image GP1 generally match, it is considered that the image GP1 generally does not match the image GA3. That is, it is considered that there is a gap between the image GA3 and the image GP1. Assuming such a case, the present disclosure will be described later by providing a first modification.

The above description will be described in the flow of a series of processes.

FIG. 15 is a flowchart showing a procedure of processing executed by the CPU 210.

In step S1, the CPU 210 controls the image pickup device 300 to cause the image pickup device 300 to take a picture of a part of the top plate 12 (mounting table). For example, the CPU 210 transmits a shooting instruction signal (control signal) to the image pickup apparatus 300. The image pickup apparatus 300 photographs a part of the top plate 12 according to the control signal, and transmits the image signal of the acquired image to the controller 200.

In step S2, the CPU 210 receives an image signal from the image pickup apparatus 300 and acquires the captured image.

If the image does not contain an image of the mark, the subsequent step S3 is performed.

In step S3, the CPU 210 refers to the database DB and compares the image feature amount of the acquired image with the image feature amount of each image stored in the memory 220. Subsequent step S4 is a process when the comparison results match.

In step S4, the CPU 210 reads the positional relationship data associated with the matching images from the database DB.

In step S5, the CPU 210 uses the read data to generate and output a control signal for changing the position of the image pickup device 300 so that the mark is placed in the field of view of the image pickup device 300. In step S6, the CPU 210 changes the position of the image pickup apparatus 300 according to the output control signal, and then controls the image pickup apparatus 300 to cause the image pickup apparatus 300 to take a picture of a part of the top plate 12.

In step S7, the CPU 210 receives the image taken again from the image pickup apparatus 300 and detects the image of the mark from the image.

According to the present embodiment, the relationship between the position representing each image other than the image including the image of the mark M and the amount of deviation from the center position of the mark M is learned by the mark position learning operation. If it is found that the same image as the image input during the later mark detection operation is included in the learned image, the amount of deviation to the center position of the mark M associated with the image is included. By moving the image pickup apparatus 300 only, the mark M can be detected more quickly and reliably.

Next, a modified example of the above-described embodiment will be described.

The first modification is the update (re-learning) of the deviation amount.

When the comparison result between the image used for learning and the image acquired by the imaging device 300 matched, the imaging device 300 was moved using the positional deviations Δx and Δy acquired from the database DB. The misalignments Δx and Δy are determined as positions where the mark M is detected at a predetermined position in the acquired image, for example, the center of the field of view, when the image pickup device 300 is moved and then photographed again. Therefore, for example, the mark M should be detected at the center of the field of view of the image pickup apparatus 300. However, in reality, the mark M is rarely detected in the center of the field of view. The reason is that the range of the image reacquired by the image pickup apparatus 300 rarely matches the range of each image in the training data used at the time of learning. Therefore, the center of the field of view of the image pickup apparatus 300 after movement usually does not coincide with the center point O of the mark M.

FIG. 16 shows an example in which the center C of the field of view of the image pickup apparatus 300 after movement and the center point O of the image GM of the mark do not match. In FIG. 16, the deviation between the two is shown by two differences (−δx) and (−δy), respectively. The reference numerals mean that the position of the center point O exists in the negative directions of the illustrated X-axis and Y-axis when viewed from the center C of the visual field.

The CPU 210 of the controller 200 acquires the difference (-δx) and (-δy) in order to cancel the deviation between the two, and uses the acquired difference (-δx) and (-δy) to obtain the initial positional deviation Δx and (-δy). Update Δy. That is, the CPU 210 uses the acquired differences (-δx) and (-δy) to update the positional deviations Δx and Δy, which are positional relationship data indicating the relationship between the position represented by each image in the database DB and the position of the mark M. do.

FIG. 17 shows the new misalignments Δx'and Δy'after updating the misalignments Δx and Δy with the differences (−δx) and (−δy). The CPU 210 updates the initial positional deviations Δx and Δy by the operations of Δx ′ = Δx + (−δx) and Δy ′ = Δy + (−δy). When the image pickup apparatus 300 is moved using Δx'and Δy', the center of the field of view of the image pickup apparatus 300 coincides with the center point O of the mark M. The CPU 210 updates the amount of deviation of the database DB of the image GA3 for moving from the position where the original image GA3 was obtained to the mark M.

FIG. 18 shows an example of the updated database DB for image GA3. Comparing with the database DB for the image GA3 before the update shown in FIG. 9, it is understood that each value of the deviation amount is updated. The description in the column of image feature amount is omitted.

According to the above processing, the deviation amount obtained by the mark position learning operation is updated according to the result of the mark detection operation. Since the database DB can be updated according to the actual operating environment, the mark detection operation can be executed quickly thereafter.

In the first modification, it is assumed that the mark M is present in the field of view of the image pickup device 300 after the image pickup device 300 is moved. However, there may be cases where the mark M does not exist in the field of view. The CPU 210 changes the position of the image pickup apparatus 300 according to the positional relationship data associated with the images in the database DB in which the comparison results match, and then re-takes the image of the mark M from the image taken by the image pickup apparatus 300. Is determined whether or not can be detected. Then, when the CPU 210 cannot detect the image of the mark M, the CPU 210 acquires an image while changing the position of the image pickup apparatus 300 in a predetermined order, and whether the image of the mark M exists in the acquired image. Judge whether or not. For example, the CPU 210 may perform an operation of detecting the mark M from that position by a brute force detection method. More specifically, the CPU 210 acquires an image while changing the position of the image pickup apparatus 300 in a predetermined order, and performs an operation of determining whether or not an image of the mark M exists in the acquired image. Then, the operation of detecting the mark M may be performed. Normally, it is considered that the mark M is present at a position relatively close to the field of view, although it is not in the field of view. Therefore, even if the brute force detection method is adopted, the CPU 210 can detect the mark M sufficiently quickly. Then, when the mark M can be detected, the data to which the erroneous deviation amount is given may be updated. The updated data is generated by adding the direction and distance moved by the brute force detection operation to the initial deviation amount. The CPU 210 may update the initial deviation amount stored in the database DB with the new deviation amount generated.

On the other hand, it is conceivable that the amount of deviation becomes too large even though the database DB for each image is correct because it is determined that the two images are erroneously matched. Therefore, if the mark M does not exist in the field of view after the image pickup apparatus 300 is moved, the database DB may be maintained without being updated.

Next, a second modification will be described.

The second modification is the introduction of priority data indicating the priority of each image. The priority data is an additional parameter stored in the database DB in association with the image feature amount of each image for a plurality of images that do not include the image of the mark M. The priority indicates to some extent that the image feature amount used when comparing two images is useful in determining the agreement between the two images. In other words, the priority indicates the degree of ease of matching of the comparison results when the image feature amounts are compared using the image feature amounts associated on the database DB. Hereinafter, the priority and the database DB thereof will be described with reference to the image GA3 shown in FIG. 9 as an example.

FIG. 19 shows an example of a database DB for image GA3 in which priority is introduced. In the database DB, in addition to the image feature amount for each partial image, the deviation amount (positional relationship data) indicating the relationship between the position represented by each image and the mark position, the priority indicating the "priority" of each partial image. The data is related.

The significance of the priority will be described with reference to FIG. 7. For example, the partial images 1-1 to 1-4 are all dark in color as a whole, and the edges, which are the image feature amounts of the present embodiment, are seemingly small. Therefore, it is considered that the image feature amounts of these partial images are not useful in determining whether or not they match the image feature amounts of other images acquired from the image pickup apparatus 300 during the mark detection operation. On the other hand, for example, the partial image 3-1 has a plurality of characteristic edges in terms of shape and position, and is therefore useful in determining whether or not the partial image matches the image feature amount of the other image. Conceivable. The priority is a parameter that reflects such circumstances.

When the priority is set, when it is determined that there are a plurality of partial images determined to match the image feature amounts over one or a plurality of images, the image pickup apparatus 300 uses the amount of deviation of which partial image. It is especially useful in deciding whether to move. Specifically, the CPU 210 compares the image feature amount of the acquired partial area of the image with the image feature amount of the partial area of each image stored in the memory 220. When the CPU 210 determines that the comparison results match for the plurality of images stored in the memory 220, the CPU 210 converts the plurality of images into a specific image according to the magnitude of the priority of each of the plurality of images. Read the associated positional relationship data. More specifically, the positional relationship data associated with a specific image having the highest priority is read out. The CPU 210 moves the image pickup apparatus 300 by utilizing the read-out deviation amount. As a result, the image pickup apparatus 300 can be moved to the position where the mark exists more reliably. In the above description of the process, the image feature amount of the partial region of the acquired image is illustrated, but the image feature amount may be obtained from the entire acquired image.

The priority may be fixed or updated. An example of updating will be described below.

For example, when the CPU 210 determines that an image feature amount matching the image feature amount of the partial image 3-1 exists in another image, the CPU 210 moves the image pickup device 300 according to the deviation amount associated with the partial image 3-1 to position the image feature amount. To change. After changing the position, the CPU 210 that has acquired a new image from the image pickup apparatus 300 determines whether or not the image GM of the mark actually exists in the image. When the image of the mark M is detected from the new image, it can be said that the amount of deviation associated with the partial image 3-1 was correct. That is, it can be said that the image feature amount of the partial image 3-1 was useful in the match determination. Therefore, when the image of the mark M is detected from the new image, the CPU 210 updates the priority associated with the partial image 3-1 to be higher. For example, in the example of FIG. 19, the priority may be increased by 1. Since the priority can be updated according to the actual operating environment, the CPU 210 can quickly execute the mark detection operation.

The initial value of the priority may be appropriately set by the user of the mark detection system 1000, or a uniform value, for example, priority 1, may be given at the beginning.

In the above description, the priority is described in units of partial images, but the priority may be set in units of one image. For example, in FIG. 4, when it can be said that there is no image feature amount useful for identification inside the top plate 12 not including the edge, it is considered sufficient to set the priority of the entire image.

The amount of misalignment described so far is expressed as a difference between the X-axis direction and the Y-axis direction as shown in FIG. 7, but the expression method is an example. As another example, a polar coordinate system using angles and distances may be adopted.

In the example of FIG. 1, the image pickup apparatus 300 is attached to the end effector 130 in the present embodiment, but the configuration is an example. The image pickup apparatus 300 may be provided at another position. For example, the image pickup apparatus 300 may be mounted on another AGV (second transport vehicle) different from the AGV 10 (first transport vehicle). When the article to which the mark M is given is the mounting table, that is, the top plate of the first transport vehicle for transporting the work W, the imaging device on the second transport vehicle photographs the top plate of the first transport vehicle and marks the mark. Can be detected. Further, the second carrier may be equipped with an articulated robot in addition to the imaging device. That is, the second transport vehicle may include an articulated robot 100 having an image pickup device 300 as shown in FIG. When such a second carrier is used, the CPU 210 of the controller 200 can detect the image of the mark and operate the articulated robot according to the detection result. As a result, at an arbitrary position, the controller 200 can operate the articulated robot 100 according to the detection result of the mark image, and can pick up and move the work W.

As described above, the processing performed by the mark detection system 1000 is realized by the CPU 210 executing the computer program stored in the memory 220. Such a computer program is a set of instructions for causing the CPU 210 to execute the processing procedure (method) shown in FIG. 15, recorded on a recording medium such as a CD-ROM, and distributed as a product on the market, or It can be transmitted through a telecommunication line such as the Internet. Further, the CPU 210 and the memory 220 in which the computer program is stored may be realized as a signal processing circuit such as a DSP (Digital Signal Processor) in which the computer program is incorporated in one semiconductor circuit. That is, such a DSP can function as the controller 200 of FIG.

The mark detection system of the present disclosure detects marks using captured images in factories, warehouses, construction sites, logistics, hospitals, etc., positions them, and moves and transports workpieces such as cargo and parts. Can be suitably used for.

10 ... Unmanned carrier (AGV), 12 ... Top plate, 100 ... Articulated robot, 200 ... Controller, 210 ... CPU, 220 ... Memory, 300 ... Imaging device , 1000 ... mark detection system, DB ... database, GM ... mark image, M ... mark, W ... work

Claims (13)

A mark detection system that detects marks using an image pickup device, which is an image pickup device that has a field of view that allows partial imaging of an article with a mark, and a storage device that stores a database acquired by the mark position learning operation. The database is used for each image of a plurality of images that do not include the image of the mark among the plurality of images obtained by photographing different parts of the article in a plurality of times by the imaging device. A storage device and a signal processing device that associate at least one image feature amount of the image and positional relationship data indicating the relationship between the position represented by each image and the position of the mark are provided, and during mark detection operation. In the signal processing apparatus, when a part of the article is photographed by the imaging apparatus and the image acquired from the imaging apparatus does not include an image of the mark, the signal processing apparatus obtains the data by referring to the database. The image feature amount of the image is compared with the image feature amount of each image stored in the storage device, and if the comparison results match, the positional relationship associated with the matching image is found from the database. The data is read out, and the read out positional relationship data is used to generate and output a control signal for changing the position of the image pickup device so that the mark is placed in the field of view of the image pickup device, and the position is changed. A mark detection system that detects an image of the mark from an image taken again by using the image pickup device. The storage device holds feature data related to the image of the mark, and the signal processing device refers to the feature data and determines whether or not the image of the mark is included in the image acquired from the imaging device. The mark detection system according to claim 1, wherein the mark detection system determines whether or not. The second aspect of the present invention, wherein when the signal processing device determines that the image acquired from the imaging device contains an image of the mark, the signal processing device executes a predetermined positioning process using the detection position of the mark. The mark detection system described. In the database, for a plurality of images that do not include the image of the mark, in addition to at least one image feature amount of each image, the positional relationship data indicating the relationship between the position represented by each image and the position of the mark, and each image. The priority data indicating the priority of is related, and the priority is the degree of ease of matching of the comparison results when the image feature amounts are compared using the related image feature amounts. The signal processing device compares the acquired image feature amount of the image with the image feature amount of each image stored in the storage device, and a plurality of image feature amounts stored in the storage device. If it is determined that the comparison results of the images match, the positional relationship data associated with the specific image is read out according to the magnitude of the priority of each of the plurality of images. The mark detection system according to any one of 1 to 3. The signal processing device changes the position according to the positional relationship data associated with the specific image, and then detects the image of the mark from the image taken again by the imaging device. The mark detection system according to claim 4, wherein the priority data is updated so that the priority associated with the specific image is higher. The positional relationship data, which is held by the database and shows the relationship between the position represented by each image and the position of the mark, indicates the amount of misalignment from the position represented by each image to the position of the mark. The position of the mark is determined as a position where the mark is detected at a predetermined position in the acquired image when the image is taken by the imaging device, and the signal processing device changes the position after the mark is changed. When the image of the mark is detected from the image taken again by the image pickup device and the position where the image of the mark is detected does not match the predetermined position in the acquired image. , The difference between the position where the image of the mark is detected and the predetermined position is acquired, and the difference is used to obtain the positional relationship data indicating the relationship between the position represented by each image of the database and the position of the mark. The mark detection system according to any one of claims 1 to 5, which is to be updated. The signal processing device changes the position according to the positional relationship data associated with the specific image, and then detects the image of the mark from the image taken again by the imaging device. If this is not possible, the image is acquired while changing the position of the imaging device in a predetermined order, and it is determined whether or not the image of the mark exists in the acquired image, according to claim 1. The mark detection system according to any one of 6. At least one image feature amount of each image held by the database is a plurality of image feature amounts of each image, and each of the plurality of image feature amounts is a plurality of partial images constituting each image. The signal processing device refers to the database, and the image feature amount of each partial image of each image stored in the storage device is the image feature amount of the acquired image. The mark detection system according to any one of claims 1 to 7, wherein it is determined whether or not the image is included, and when the degree of inclusion matches a predetermined condition, it is determined that the comparison results match. The article to which the mark is given is a mounting table of a first transport vehicle for transporting a work, and the image pickup device is provided on a second transport vehicle different from the first transport vehicle, according to claim 1. 8. The mark detection system according to any one of 8. The mark detection system according to claim 9, wherein the second transport vehicle includes an articulated robot, and operates the articulated robot according to a detection result of an image of the mark by the signal processing device. It is a signal processing circuit used in a mark detection system that detects using an image pickup device, and the mark detection system includes an image pickup device having a field of view capable of partially photographing an article with a mark, and a mark position learning operation. A storage device that stores the database acquired by the above, and the database is an image of the mark among a plurality of images obtained by photographing different parts of the article in a plurality of times by the imaging device. It is provided with a storage device that associates at least one image feature amount of each image and positional relationship data indicating the relationship between the position represented by each image and the position of the mark for a plurality of images that do not include. The signal processing circuit causes the image pickup device to take a picture of a part of the article during the mark detection operation, and when the image acquired from the image pickup device does not include the image of the mark, the database is used. With reference, the acquired image feature amount of the image is compared with the image feature amount of each image stored in the storage device, and if the comparison results match, the matching image is obtained from the database. The associated positional relationship data is read out, and the read positional relationship data is used to generate a control signal for changing the position of the imaging device so that the mark is placed in the field of view of the imaging device. A signal processing circuit that outputs an image, changes the position, and then detects an image of the mark from an image taken again using the imaging device. A computer program for operating a signal processing circuit used in a mark detection system that detects a mark using an image pickup device. The mark detection system provides a field of view in which a marked article can be partially photographed. An image pickup device and a storage device that stores a database acquired by a mark position learning operation. The database is a plurality of storage devices obtained by photographing different parts of the article in a plurality of times by the image pickup device. Of the images, for a plurality of images that do not include the image of the mark, at least one image feature amount of each image and positional relationship data indicating the relationship between the position represented by each image and the position of the mark are related. It is equipped with a storage device attached to it, and during the mark detection operation, the computer program causes the signal processing circuit to control the image pickup device to take a picture of a part of the article, and obtains the data from the image pickup device. When the image does not include the image of the mark, the database is referred to and the acquired image feature amount of the image is compared with the image feature amount of each image stored in the storage device for comparison. When the results match, the positional relationship data associated with the matched image is read from the database, and the read positional relationship data is used to put the mark in the field of view of the imaging device. , A computer program that generates and outputs a control signal for changing the position of the image pickup device, changes the position, and then detects an image of the mark from an image taken again using the image pickup device. .. A method executed by using a mark detection system for detecting a mark using an image pickup device, wherein the mark detection system includes an image pickup device having a field of view capable of partially photographing an article to which the mark is given, and a mark. A storage device that stores a database acquired by a position learning operation, wherein the database is one of a plurality of images obtained by photographing different parts of the article in a plurality of times by the imaging device. A storage device that associates at least one image feature amount of each image and positional relationship data indicating the relationship between the position represented by each image and the position of the mark for a plurality of images that do not include the image of the mark. And a signal processing device, and during the mark detection operation, the image pickup device is made to photograph a part of the article, and the signal processing circuit includes an image of the mark in the image acquired from the image pickup device. When the image acquired from the image pickup apparatus does not include the image of the mark, the signal processing circuit refers to the database to obtain the image feature amount of the acquired image. The image feature amount of each image stored in the storage device is compared, and if the comparison results match, the database is made to read out the positional relationship data associated with the matched image, and the above-mentioned Using the positional relationship data, the signal processing circuit generates and outputs a control signal for changing the position of the imaging device so that the mark is placed in the field of view of the imaging device, and changes the position. After that, a method of causing the signal processing circuit to detect an image of the mark from an image taken again by using the imaging device.

Images