WO2023112259A1

WO2023112259A1 - Image processing device

Info

Publication number: WO2023112259A1
Application number: PCT/JP2021/046526
Authority: WO
Inventors: 康樹野上
Original assignee: 株式会社Fuji
Priority date: 2021-12-16
Filing date: 2021-12-16
Publication date: 2023-06-22

Abstract

This image processing device comprises: a process execution unit that executes an inspection process for inspecting the state of a camera that is utilized in a mounting step for mounting a component on a substrate, the inspection process including a process for causing the camera to image a predetermined reference plate different from the component and the substrate; and a notification unit that executes a predetermined notification operation if an out-of-reference region having an area greater than or equal to a predetermined area is present in an image captured by the camera in the inspection process, the out-of-reference region being a blob region in which a plurality of pixels are present, the plurality of pixels being connected and having pixel values of which a difference from a pixel value indicating the reference plate in the image is greater than or equal to a predetermined value.

Description

Image processing device

This specification relates to technology for processing images from cameras used in the mounting process of mounting components on boards.

Patent Document 1 discloses a control device that processes an image obtained by a camera used in a component mounter. The control device identifies an area in which normal imaging cannot be performed in the entire area of the image as an unimaging area, and holds the part using nozzles that are scheduled to appear in the identified unimaging area among the plurality of nozzles. cancel. Even if a problem occurs in the camera, another nozzle is used to continue the mounting process of mounting components on the board.

JP 2018-098280 A

In the above patent document, a rectangular area that circumscribes the dropped part area generated by dropping the part onto the camera in the vertical direction and the horizontal direction is defined as the non-imaging area. For this reason, the area of the non-imaging area is larger than the area of the dropped part area, and the malfunction of the camera cannot be evaluated correctly. Further, the above patent document is a technique for continuing the mounting process even if a problem occurs in the camera, and is not a technique for evaluating the camera problem and notifying the user of the camera problem. Techniques are provided herein for evaluating camera defects and for notifying users of such defects.

An image processing apparatus disclosed in the present specification is a processing execution unit that executes inspection processing for inspecting the state of a camera used in a mounting process for mounting a component on a board, wherein the inspection processing includes the component and the The processing execution unit includes processing for causing the camera to image a predetermined reference plate different from the substrate, and the image captured by the camera in the inspection processing includes a non-reference area having a predetermined area or more. a notifying unit for executing a predetermined notifying operation, wherein the non-reference area includes a plurality of pixel values having a difference of a predetermined value or more from a pixel value indicating the reference plate in the image. and the reporting unit, which is a blob area in which pixels are connected.

Camera malfunctions include, for example, dust adhering to the camera lens and malfunction of camera parts. Such defects may be imaged in a color different from the color of the object being imaged. Alternatively, in the imaged image, the luminance as a pixel value may differ between the defect portion and the imaged object. In addition, if the area of the region corresponding to the defect in the image is relatively large, there is a possibility that the imaging of the target will not be performed normally and the mounting process will fail. The non-reference region is a region having a color different from that of the reference plate and having a predetermined area or more. If there is a non-reference area in the image, it can be evaluated that the camera is malfunctioning. According to the above configuration, the notification operation is performed when it can be evaluated that the camera is malfunctioning. The user can be notified of camera malfunctions.

The side view which expresses the structure of a mounting apparatus typically is shown. 1 shows a block diagram of a mounting device; FIG. 4 shows a flowchart of first camera inspection processing according to the first embodiment. 4 shows a flowchart of second camera inspection processing according to the first embodiment. FIG. 11 shows a flowchart of a first camera inspection process according to a second embodiment; FIG.

The main features of the embodiments described below are listed. It should be noted that the technical elements described below are independent technical elements, exhibiting technical usefulness alone or in various combinations, and are limited to the combinations described in the claims as filed. not a thing

(Feature 1) The notification operation may not be executed when the non-reference area does not exist in the image.

According to such a configuration, unnecessary execution of the notification operation can be suppressed.

(Feature 2) The reference plate may be a white plate.

(Feature 3) The predetermined area may be equal to or less than the area of the smallest part.

Images of dust adhering to camera lenses and images showing malfunctions of camera parts are usually smaller than images showing parts. According to the above configuration, it is possible to appropriately determine a blob area that is equal to or smaller than the area of the smallest component as a non-reference area.

(Feature 4) The camera can capture an image of a nozzle that picks up the component and conveys it to the board, and in the inspection process, the reference plate may be picked up by the nozzle.

According to such a configuration, it is possible to inform the user of a malfunction of the camera capable of imaging the nozzle.

(Feature 5) In the image, in the central area indicating the center of the lens of the camera, when the blob area having the first area or more, which is the predetermined area, exists, the non-standard area in the image is determined to exist, and in the image, in the peripheral area indicating the peripheral edge of the lens of the camera, the second area that is the predetermined area and is equal to or larger than the second area that is smaller than the first area It may be determined that the non-reference area exists in the image when the blob area of .

According to such a configuration, even if the image indicating dust or failure at the peripheral edge of the lens is smaller than the image indicating dust or failure at the center of the lens, the execution of the notification operation allows the image indicating dust or failure at the peripheral edge of the lens. The user can be prompted to deal with the debris or malfunction of the

(Feature 6) The camera may capture an image of the substrate supported on a predetermined support, and the reference plate may be supported by the support during the inspection process.

According to such a configuration, it is possible to notify the user of a malfunction of the camera capable of imaging the substrate supported on the support base.

(First embodiment)
(Configuration of mounting apparatus 10; FIGS. 1 and 2)
The mounting apparatus 10 is an apparatus capable of executing a mounting process for mounting components on a board. A component is an electronic component such as a resistor. The mounting apparatus 10 includes a moving device 12 , a nozzle 14 , a display 16 , a bottom parts camera 20 , a side parts camera 22 , a mark camera 24 , a control section 30 , a support base 50 and a parts feeder 60 . The nozzle 14 , side parts camera 22 and mark camera 24 are attached to the moving device 12 .

Each unit 12-24, 60 is controlled by the control unit 30. The control unit 30 has a CPU 32 and a memory 34 . The CPU 32 can execute various processes according to programs 40 stored in the memory 34 . The CPU 32 can, for example, analyze the images captured by each

camera

20, 22, 24. FIG.

The support base 50 is a base that supports a substrate that is transported by a transport device (for example, a conveyor, not shown). The component feeder 60 is a device that supplies components into the mounting apparatus 10 .

The nozzle 14 is a device that picks up the parts supplied by the parts feeder 60 . The display 16 is a device capable of displaying various information.

The moving device 12 is a device that can move in the horizontal direction (horizontal direction and orthogonal direction to the paper surface). The moving device 12 moves above the component feeder 60, for example. The nozzle 14 of the moving device 12 picks up the component supplied by the component feeder 60 . The moving device 12 moves from the upper side of the component feeder 60 to the upper side of the support base 50 after the nozzle 14 picks up the component. The nozzle 14 mounts the component sucked by the nozzle 14 on the board supported by the support table 50 . This implements the mounting process.

The bottom parts camera 20 is arranged between the parts feeder 60 and the support base 50 . The lower surface parts camera 20 can image the nozzle 14 moving between the parts feeder 60 and the support base 50 from below. By analyzing the image captured by the lower part camera 20, the position within the nozzle 14 of the part sucked by the nozzle 14 can be obtained. Based on the position, it is inspected whether or not the component is correctly sucked by the nozzle 14 .

The side parts camera 22 is arranged adjacent to the nozzle 14 . The side part camera 22 can image the part sucked by the nozzle 14 from the side. By analyzing the image captured by the side part camera 22, the orientation of the part sucked by the nozzle 14 can be obtained. Based on the posture, it is inspected whether or not the component is correctly sucked by the nozzle 14 .

The mark camera 24 is arranged adjacent to the nozzle 14 . The mark camera 24 can image the substrate supported by the support table 50 from above. By analyzing the image captured by the mark camera 24, the appearance of the board on which the components are mounted is inspected. Also, by analyzing the image captured by the mark camera 24, the board identification number can be obtained from the mark (for example, bar code) printed on the board.

The images captured by each camera 20-24 are monochrome. Note that in a modified example, the images captured by each of the cameras 20-24 may be in color.

Dirt such as dust may adhere to the lens of the camera such as the bottom part camera 20 . Dust adhering to the lens can usually be removed by a blower (not shown) of the mounting device 10 . However, a situation is assumed in which the dust attached to the lens cannot be completely removed by the blower. Dust that cannot be removed by the blower is manually removed by the user of the mounting apparatus 10, for example.

In this embodiment, the CPU 32 performs the first camera inspection process (see FIG. 3) and the second camera inspection process (see FIG. 4) in accordance with the program 40 to prompt the user to remove dust adhering to the lens. Execute.

(First camera inspection process; FIG. 3)
The first camera inspection process is a process for inspecting whether or not dust adheres to the bottom parts camera 20 . For example, the first camera inspection process is performed before starting the mounting process. For example, the first camera inspection process is triggered by an instruction from the user. Note that, in a modified example, the first camera inspection process may be performed during the mounting process. Also, the first camera inspection process may be automatically executed without an instruction from the user.

In S10, the CPU 32 causes the nozzle 14 to pick up the reference plate 100 (see FIG. 1). The reference plate 100 is a white plate. The reference plate 100 is, for example, prepared in advance at a predetermined position (for example, next to the component feeder 60) inside the mounting apparatus 10. As shown in FIG. The CPU 32 moves the moving device 12 to the upper side of a predetermined position, and causes the nozzle 14 to absorb the reference plate 100 arranged at the predetermined position.

In S12, the CPU 32 moves the moving device 12 to the upper side of the lower parts camera 20 and causes the lower parts camera 20 to image the reference plate 100 sucked by the nozzle 14 . Thereby, the CPU 32 acquires an image (hereinafter referred to as a “first processed image”) showing the reference plate 100 from the lower parts camera 20 . For example, when dust is attached to the lens of the lower surface parts camera 20, as shown in FIG. included. That is, when dust adheres to the lens of the bottom parts camera 20, in the first processed image, a pixel having a luminance as a pixel value indicating the white color of the reference plate 100 and a pixel value indicating the reference plate 100 There exists a pixel having a luminance as a pixel value different from that of . A blob region composed of pixels having pixel values different from those representing the reference plate 100 indicates dust adhering to the lens of the bottom part camera 20 . On the other hand, when dust does not adhere to the lens of the bottom parts camera 20, the blob area is not included in the first processed image. A blob area is an area in which a plurality of pixels having pixel values indicating a color (or brightness) different from the pixel values indicating white of the reference plate 100 (for example, “225 to 255”) are connected.

In S14, the CPU 32 determines whether the pixel value of each of the plurality of pixels forming the first processed image is smaller than a predetermined threshold value (eg "220"). A pixel value smaller than a predetermined threshold means that the color of the pixel indicated by the pixel value is different from the white color of the reference plate 100 . After completing the above determination for each of the plurality of pixels forming the first processed image, the CPU 32 proceeds to S16.

At S16, the CPU 32 uses the result of determination at S14 (see the graph at the bottom of the page) to determine whether or not the first processed image includes one or more blob areas. Here, whether or not two pixels having pixel values indicating different colors are connected is determined, for example, by using a number that identifies each pixel (for example, the number indicated on the horizontal axis of the graph at the bottom of the page). and is judged. If the CPU 32 determines that one or more blob areas are included in the first processed image (YES in S16), the process proceeds to S18 and subsequent steps. On the other hand, when the CPU 32 determines that the first processed image does not include the blob area (NO in S16), the CPU 32 skips the processes after S18 and ends the process of FIG.

In S18, the CPU 32 determines whether or not the area of at least one blob area among the one or more blob areas is larger than a predetermined value. Here, the predetermined value is set to an area equal to or smaller than the smallest component among various types of components supplied by the component feeder 60 . The image of dust on a camera lens is usually smaller than the image showing the part. According to such a configuration, it is possible to appropriately determine a blob region that is equal to or smaller than the area of the smallest component as a region indicating dust adhering to the lens. It should be noted that, in a modified example, the predetermined value may be set independently of the sizes of various types of parts supplied by the parts feeder 60 .

When the CPU 32 determines that the area of at least one blob area is larger than the predetermined value (YES in S18), the process proceeds to S20. In S<b>20 , the CPU 32 causes the display 16 to display an instruction to remove dust adhering to the lens of the bottom parts camera 20 . As a result, it is possible to evaluate whether dust has adhered to the lens of the bottom part camera 20 and notify the user of the dust adherence.

On the other hand, if the CPU 32 determines that the area of all of the one or more blob areas is equal to or less than the predetermined value (NO in S18), the CPU 32 proceeds to S22. A blob area having an area smaller than a predetermined value is highly likely to be, for example, noise from the bottom part camera 20 rather than dust adhering to the lens. In S22, the CPU 32 corrects a plurality of pixel values representing a plurality of pixels forming the blob region in order to remove noise from the bottom part camera 20. FIG. For example, the CPU 32 corrects the plurality of pixel values using the average value of the plurality of pixel values. When S20 or S22 ends, the CPU 32 ends the processing of FIG.

Note that if it is determined that the area of each of the one or more blob areas is smaller than the predetermined value (NO in S18), the process of S20 is not executed. Unnecessary execution of the process of S20 (that is, notification to the user) can be suppressed.

(Second camera inspection process; FIG. 4)
The second camera inspection process is a process for inspecting whether or not dust adheres to the mark camera 24 . For example, the trigger for the second camera inspection process is similar to the trigger for the first camera inspection process. The second camera inspection process is the same as the first camera inspection process except that the processes of S100 and S102 are executed instead of S10 and S12 of FIG.

At S100, the CPU 32 causes the transport device to transport the white reference plate 200, which is the same as the reference plate 100. Thereby, the reference plate 200 is supported by the support base 50 . The reference plate 200 is, for example, put into the transport device by a user.

In S102, the CPU 32 moves the moving device 12 to the upper side of the support base 50, and causes the mark camera 24 to image the reference plate 200 supported by the support base 50. As a result, the CPU 32 acquires an image representing the reference plate 200 from the mark camera 24 (hereinafter referred to as a “second processed image”).

When S102 ends, the CPU 32 executes the processes of S14 to S22. The second processed image is used in S14 to S22 of the second camera inspection process. For example, when the CPU 32 determines that the area of at least one blob area included in the second processed image is larger than a predetermined value (YES in S18), the CPU 32 instructs to remove dust adhering to the lens of the mark camera 24. is displayed on the display 16 (S20). Accordingly, it is possible to evaluate the presence of dust on the lens of the mark camera 24 and notify the user of the presence of dust.

(correspondence relationship)
The control unit 30 of the mounting apparatus 10 is an example of the "image processing apparatus". In the first camera inspection process of FIG. 3, the lower part camera 20, the reference plate 100, and the predetermined value in S18 of FIG. be. In the first camera inspection process, the CPU 32 that executes the processes of S10 to S18 in FIG. 3 and the CPU 32 that executes the process of S20 in FIG. In the second camera inspection process of FIG. 4, the mark camera 24, the reference plate 200, and the predetermined value in S18 of FIG. . In the second camera inspection process, the CPU 32 that executes the processes of S100, S102, and S14 to S18 in FIG. 4 and the CPU 32 that executes the process of S20 in FIG. An example.

(Second embodiment)
This embodiment is the same as the first embodiment, except that the content of the first camera inspection process for inspecting whether or not dust adheres to the bottom parts camera 20 is different.

(First camera inspection process; FIG. 5)
S10 to S16 are the same as in the first embodiment. When the CPU 32 determines that the first processed image does not include the blob area (NO in S16), the CPU 32 skips the processing after S200 described later and ends the processing in FIG.

When the CPU 32 determines that one or more blob areas are included in the first processed image (YES in S16), the process proceeds to S200. In S200, the CPU 32 selects one blob area (hereinafter referred to as "target blob area") from one or more blob areas.

In subsequent S202, the CPU 32 determines whether or not the target blob area is included in the central area including the center of the lens of the bottom parts camera 20 in the first processed image. Here, the position of the target blob area is determined, for example, by using the numbers that identify the pixels forming the target blob area (for example, the numbers indicated on the horizontal axis of the graph at the bottom of the page of FIG. 3). be done. When the CPU 32 determines that the target blob area is included in the central area (YES in S202), the process proceeds to S204. Whether each pixel in the first processed image belongs to the central region or the peripheral region is predetermined and stored in the memory 34 . In addition, when a plurality of pixels constituting the target blob area straddles the central area and the peripheral area, the area including the larger area of the blob area or the area including the larger number of pixels of the blob area may be determined as an area including the blob area.

In S204, the CPU 32 determines whether or not the area of the target blob area is larger than the first predetermined value. When the CPU 32 determines that the area of the target blob region is larger than the first predetermined value (YES in S204), the process proceeds to S210. In S<b>210 , the CPU 32 decides to display an instruction to remove dust adhering to the lens of the bottom parts camera 20 . Note that the display of the instruction is not performed until the determinations of S202 and S204 are performed for all of one or more blob areas.

Also, when the CPU 32 determines that the area of the target blob area is equal to or less than the first predetermined value (NO in S204), the process proceeds to S212. S212 is the same as S22 in FIG. After completing S210 or S212, the CPU 32 proceeds to S230.

Also, when the CPU 32 determines that the target blob area is not included in the central area but is included in the peripheral area indicating the peripheral edge of the lens of the bottom part camera 20 (NO in S202), the process proceeds to S214. In S214, the CPU 32 determines whether or not the area of the target blob area is larger than the second predetermined value. Here, the second predetermined value is a value smaller than the first predetermined value of S204. When the CPU 32 determines that the area of the target blob area is larger than the second predetermined value (YES in S214), the process proceeds to S220. S220 is similar to S210.

Also, when the CPU 32 determines that the area of the target blob area is equal to or less than the second predetermined value (NO in S214), the CPU 32 proceeds to S222. S222 is similar to S212. When S220 or S222 ends, the CPU 32 proceeds to S230.

At S230, the CPU 32 determines whether or not there is an unselected blob area among the one or more blob areas. When the CPU 32 determines that there is an unselected blob area among one or more blob areas (YES in S230), the CPU 32 returns to S200 and selects another blob area. On the other hand, when the CPU 32 determines that there is no unselected blob area among the one or more blob areas (NO in S230), the process proceeds to S232.

At S232, the CPU 32 determines whether or not it was decided at S210 or S220 to display an instruction to remove dust adhering to the lens. When the CPU 32 determines that display of the instruction has been determined (YES in S232), the process proceeds to S240. On the other hand, when the CPU 32 determines that display of the instruction has not been determined (NO in S232), the CPU 32 skips S240 and ends the processing of FIG.

S240 is the same as S20 in FIG. When S240 ends, the CPU 32 ends the processing of FIG.

In this embodiment, the second predetermined value of S214 is smaller than the first predetermined value of S204. Then, even if the blob area included in the peripheral area is smaller than the blob area included in the central area, the user is instructed to remove dust (S220). That is, even if the dust adhering to the periphery of the lens is smaller than the dust adhering to the center of the lens, the user can be prompted to remove the dust adhering to the periphery of the lens. For example, dust adhering to the center of the lens is easily removed by the blower of the mounting apparatus 10, but dust adhering to the periphery of the lens may not be completely removed by the blower. The user can be urged to remove dust that cannot be removed by the blower.

(correspondence relationship)
The CPU 32 that executes the processes of S10 to S16 and S200 to 230 in FIG. 5, and the CPU 32 that executes the process of S240 in FIG. The first predetermined value of S204 and the second predetermined value of S214 in FIG. 5 are examples of the "first area" and the "second area", respectively.

Points to note regarding the image processing apparatus described in the embodiment will be described. The “image processing device” is not limited to the control unit 30 of the mounting device 10 , and may be an external device (for example, a server) provided separately from the mounting device 10 .

The "camera state" is not limited to dust adhering to the camera lens, and may be, for example, a malfunction of camera parts. In this modified example, the "notification operation" is an operation for notifying the user of a failure of a part of the camera. For example, the user can take action to replace the failed component.

The "notification operation" is not limited to the display of the instruction in S20 of FIG.

The first camera inspection process in FIG. 3 may be employed to inspect the state of the side parts camera 22 (adhesion of dust, failure of parts, etc.). In this modified example, the side part camera 22 is an example of a "camera".

The color of the "reference plate" is not limited to white, and may be, for example, a color other than white (eg, gray).

In the first embodiment, either one of the first camera inspection process in FIG. 3 and the second camera inspection process in FIG. 4 may not be executed.

Also, the determination of the pixel value in S14 of FIG. 3 is not limited to the configuration of the above embodiment. For example, the CPU 32 may calculate a difference value between a pixel value representing the reference plate 100 and each of a plurality of pixels forming the first processed image. The CPU 32 may determine whether or not the calculated difference value is greater than or equal to a specific value that is predetermined and stored in the memory 34 . Pixel values indicating the reference plate 100 are stored in advance in the memory 34, for example. In subsequent S16, the CPU 32 may determine an area in which a plurality of pixels having pixel values with a difference value equal to or greater than a specific value are connected to each other as a blob area. In this modified example, an area in which a plurality of pixels having pixel values whose difference values are equal to or greater than a specific value are connected to each other is an example of the "non-reference area".

The technical elements described in this specification or drawings demonstrate technical usefulness either alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the techniques exemplified in this specification or drawings achieve multiple purposes at the same time, and achieving one of them has technical utility in itself.

10: Mounting Device 12: Moving Device 14: Nozzle 16: Display 20: Bottom Parts Camera 22: Side Parts Camera 24: Mark Camera 30: Control Unit 32: CPU
34: Memory 40: Program 50: Support table 60: Parts feeders 100, 200: Reference plate

Claims

A processing execution unit for executing inspection processing for inspecting a state of a camera used in a mounting process for mounting components on a board, wherein the inspection processing is performed by using a predetermined reference plate different from the component and the board as the camera. The processing execution unit including processing for causing the
A notification unit that executes a predetermined notification operation when a non-standard region, which is a region having a predetermined area or more, exists in the image captured by the camera in the inspection process, wherein the non-standard region is the The reporting unit, which is a blob region in which a plurality of pixels having pixel values whose difference from a pixel value indicating the reference plate in the image is equal to or greater than a predetermined value are connected and exist;
An image processing device comprising:
The image processing apparatus according to claim 1, wherein the notification operation is not executed when the non-reference area does not exist in the image.
The image processing apparatus according to claim 1 or 2, wherein the reference plate is a white plate.
The image processing device according to any one of claims 1 to 3, wherein the predetermined area is equal to or smaller than the area of the smallest part.
The camera is capable of capturing an image of a nozzle that picks up the component and conveys it to the substrate,
5. The image processing apparatus according to claim 1, wherein said reference plate is sucked by said nozzle in said inspection process.
determining that the non-standard area exists in the image when the blob area having the first area or more, which is the predetermined area, exists in the central area indicating the center of the lens of the camera in the image; is,
In the image, the blob area having a second area which is the predetermined area and equal to or larger than the second area smaller than the first area exists in a peripheral edge area indicating a peripheral edge of the lens of the camera. 6. The image processing apparatus according to claim 5, wherein it is determined that said non-reference region exists in said image when said non-reference region exists.
The camera is capable of imaging the substrate supported on a predetermined support base,
5. The image processing apparatus according to claim 1, wherein in said inspection process, said reference plate is supported by said support base.