JP2008033819A - Object recognition device, monitoring system, object recognition method, object recognition program, and recording medium recording the program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a monitoring system capable of specifying the kind of an object reliably. <P>SOLUTION: The monitoring system comprises a stereo camera 11 and a monitoring device 12. The monitoring device 12 comprises input image acquisition parts 41 and 42 for acquiring a photographed image from the stereo camera 11, a 3D information calculation part 50 for extracting a feature point from the acquired photographed image and calculating the spatial position of the extracted feature point, and an object recognition part 51 for recognizing an object based on the calculated spatial position of the feature point. The object recognition part 51 has a clustering part 54 for performing the clustering based on the calculated spatial position of the feature point, and a size determination part 55 for determining the object corresponding to the cluster based on the size of the cluster generated by the clustering. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an object recognition apparatus for recognizing an object from a captured image including images of one or a plurality of objects, a monitoring system using the apparatus, an object recognition method, an object recognition program, and a recording medium on which the program is recorded. It is. In particular, the present invention relates to a safety monitoring system that uses the object recognition device to prevent an industrial accident caused by an automated machine (production facility) at a production site.

  In the past, occupational accidents such as workers being involved in the above production facilities have occurred at production sites that use automated machines (production facilities) such as press machines and robot arms. As a countermeasure against this occupational accident, a safety monitoring device for monitoring whether a worker is safe is being introduced at a production site.

  In general, in order to ensure the safety of workers at the production site, safety was compromised when an operator intrusion was detected in a production facility and its surrounding area (dangerous area) where a disaster might occur. The production facility is stopped (intrusion detection), and when the presence of the worker is detected, the production facility is stopped until the safety of the dangerous area is confirmed (presence detection). Therefore, it is desirable that the safety monitoring device can realize both intrusion detection and presence detection.

  A conventional safety monitoring device uses a mat sensor, a photoelectric sensor, a laser scanner, an image sensor, and the like, and performs safety monitoring based on the detection principle of each sensor.

  For example, the mat sensor detects a change in load applied to the sensor. Therefore, in the safety monitoring device using the mat sensor, the presence detection is realized by arranging the mat sensor on the floor surface of the dangerous area so as to detect the load of the worker existing in the dangerous area.

  The photoelectric sensor detects a light shielding state between the light projector and the light receiver. Therefore, in the safety monitoring device using the photoelectric sensor, the intrusion detection is realized by arranging the photoelectric sensor at the boundary of the dangerous area so that the light shielding state is caused by the intrusion of the worker.

  In addition, the laser scanner can measure the distance between the laser light source and the reflector by measuring the propagation time of the light from when the laser beam is emitted, reflected by the reflector, and received by the reflected light. . Further, by scanning the laser beam, the measurable reflection region can be enlarged. Therefore, in a safety monitoring device using a laser scanner, the laser scanner is arranged so that the reflection object's presence area that can be measured by the laser scanner includes the danger area, and the measured distance varies depending on the presence or absence of the operator. Intrusion detection and presence detection are realized at the same time.

  In addition, the image sensor detects a visual change by monitoring an array of 2D (two-dimensional) luminance information and a time-series change. Therefore, in a safety monitoring device using an image sensor, an intrusion detection and a presence detection are realized at the same time as a laser scanner by arranging a camera so that a dangerous area is looked down on.

  Recently, a technique using a stereo camera has been proposed as an application technique of an image sensor (see, for example, Patent Document 1). According to this, since a conventional image sensor uses a 2D image captured by a single camera and measures using a 2D arrangement of luminance information and a time-series change, the influence of ambient ambient light As a result, the detection performance may be lowered, and the reliability as a safety monitoring device will be lowered.

On the other hand, when a stereo camera is used, 3D information of an object in an image can be acquired based on the principle of triangulation using parallax information between cameras. For this reason, it is generally considered to be more resistant to environmental fluctuations than conventional monocular image sensors.
International Publication No. WO01 / 039513 (published on May 31, 2001) Japanese Patent Laid-Open No. 5-50390 (published March 2, 1993) Japanese Patent Laid-Open No. 11-66319 (published on March 9, 1999)

  The information obtained from the above-described existing sensor is basically binary information that detects that a physical quantity has changed due to the presence or absence of an object. Therefore, it is impossible to distinguish whether the detected object (detected object) is a workpiece or an operator using only the above information. For this reason, conventionally, it has been determined that all detection objects are workers, and the operation of the production facility is controlled (specifically, the operation is stopped) to ensure the safety of workers.

  However, in an actual production site, it is sufficiently conceivable that an object other than an operator exists in the dangerous area such as when a work is carried into the dangerous area. Even in such a case, conventionally, since all the detected objects are judged as workers and the operation control of the production facility is performed, even when a workpiece enters, the facility stops every time, and the production efficiency is lowered. It was happening.

  In order to solve this problem, it is necessary to distinguish whether the detected object is a workpiece or an operator, that is, specify the type of the object, and control the operation of the production facility for each type of object. For example, in the safety monitoring method using the photoelectric sensor described in Patent Document 2, a detection object that does not follow the transfer cycle is determined as an operator in a production line in which workpieces are periodically transferred. By stopping the system, productivity is maintained.

  However, in the above safety monitoring method, a detection signal outside the transfer cycle that identifies the presence of the worker cannot be obtained when the worker enters the dangerous area in synchronization with the transfer cycle of the workpiece or together with the workpiece. It becomes difficult to distinguish between the work and the worker, and as a result, the operator's entry into the dangerous area is overlooked. That is, even if an operator enters the hazardous area, the production facility remains in operation, which is insufficient as a safety monitoring device.

  The present invention has been made in view of the above problems, and it is an object of the present invention to provide an object recognition device and the like that can reliably identify the type of an object.

  In order to solve the above problems, an object recognition apparatus according to the present invention is an object recognition apparatus that recognizes an object from a captured image including an image of one or a plurality of objects, an image acquisition unit that acquires the captured image, and an acquisition Feature point extracting means for extracting feature points from the captured image, spatial position calculating means for calculating the spatial position of the extracted feature points, and object recognition means for recognizing the object based on the calculated spatial position of the feature points The object recognition means comprises: clustering means for performing clustering based on the calculated spatial position of the feature points; and an object corresponding to the cluster based on the size of the cluster generated by the clustering. And an object determining means for determining.

  According to the above configuration, a captured image is acquired, feature points are extracted from the acquired captured image, a spatial position of the extracted feature points is calculated, and clustering is performed based on the calculated spatial position of the feature points. Thereby, a cluster similar to the shape of the object is generated.

  By the way, in general, an object has a different size (width, depth, height) depending on the type of the object. For example, a sufficient human being has a width and depth of about 30 to 80 cm and a height of about 150 to 180 cm. That is, the type of the object can be specified from the size of the object. Therefore, according to the present invention, the type of the object can be specified from the size of the cluster similar to the shape of the object.

  In the object recognition apparatus according to the present invention, the clustering means performs clustering with a size corresponding to a certain object, the object determining means determines whether or not the cluster corresponds to the object, and the clustering means It is preferable to perform clustering again with a size corresponding to another object with respect to feature points constituting a cluster that the object determination means determines not to correspond to the object. In this case, since clustering is repeated for each object size, the type of object corresponding to each cluster can be reliably identified.

  In the object recognition apparatus according to the present invention, the clustering unit performs clustering at a maximum size among a plurality of sizes corresponding to a plurality of objects to be recognized, and determines that the object determination unit does not correspond to the object. More preferably, clustering is performed again at the smallest size among the plurality of sizes with respect to the feature points constituting the cluster.

  In this case, feature points that could not be clustered by clustering at the maximum size or feature points that could not be clustered by clustering at the minimum size can be excluded at the initial stage of clustering iteration. The type can be identified quickly.

  Note that, as an example of a method for extracting feature points from captured images and calculating a spatial position of the extracted feature points, two captured images from a stereo camera are acquired and feature points in the two captured images are used. Then, the spatial position of the feature point is calculated. As another example, a captured image from an active sensor is acquired, information on a spatial position related to the captured image is acquired from the active sensor, and the space of the feature point is acquired based on the acquired information on the spatial position. For example, the position is calculated.

  In the object recognition device according to the present invention, the object determination means is based on the size of the cluster generated by the clustering and at least one of the existence position, the traveling direction, the moving speed, and the number related to the cluster. It is preferable to determine an object corresponding to the cluster.

  In general, there are many cases where the existence position, the traveling direction, the moving speed, and the number of objects are defined by the type of the object. For example, the work moves on the transport machine in the transport direction at a predetermined transport speed. In addition, since the weight that can be transported by the transporter is limited, the number of workpieces existing on the transporter is also limited.

  Therefore, as described above, not only the size of the cluster similar to the shape of the object but also the existence position, the traveling direction, the moving speed, and the number of the cluster are used to correspond to the cluster. By determining the object, the type of the object can be more reliably specified.

  Note that the probability of object determination differs depending on the size, existence position, traveling direction, moving speed, and number of clusters. Therefore, the object determination unit performs statistical processing on the object determination based on the size of the cluster and the object determination based on at least one of the existence position, the traveling direction, the moving speed, and the number related to the cluster. More preferably, an object corresponding to the cluster is determined.

  In the object recognition apparatus according to the present invention, the surface of the object is provided with a mark capable of identifying the object, and the object determination unit is configured to capture the captured image corresponding to the spatial position of the cluster generated by the clustering. The mark may be extracted from the region, and an object corresponding to the cluster may be determined based on the extracted mark and the size of the cluster. In this case, the type of the object can be more reliably identified by determining the object corresponding to the cluster using the size of the cluster and the mark. The spatial position of the cluster may be a spatial region occupied by the cluster, or may be the center of gravity of the cluster.

  The object recognition apparatus according to the present invention further includes detection signal acquisition means for acquiring a detection signal from a sensor for detecting an object, wherein the object determination means determines the object based on the size of the cluster and acquires the detection signal. An object corresponding to the cluster may be determined based on detection of the object by the means.

  Here, examples of the sensor that detects an object include an infrared camera that can acquire temperature information of the object, a mat sensor that can detect the load of the object, and the like.

  In the above case, since the object corresponding to the cluster is determined based on the determination of the object based on the size of the cluster and the detection of the object by the sensor, the type of the object can be more reliably specified.

  Storage means for storing the history information of the spatial position of the cluster generated by the clustering; and estimation means for estimating a spatial region where the cluster is next located based on the history information of the spatial position of the cluster; It is preferable to further comprise cluster integration means for extracting feature points included in the spatial region of the cluster estimated by the estimation means from the feature points extracted by the feature point extraction means and integrating them into the clusters. In this case, the spatial region of the current cluster is estimated from the spatial position of the past cluster, and the feature points included in the estimated spatial region are integrated into the cluster in advance, so these feature points can be excluded from the clustering target. As a result, the clustering process can be speeded up.

  In order to solve the above problems, a monitoring system according to the present invention is a monitoring system that monitors a predetermined area, and includes an imaging unit that performs imaging of the predetermined area, and an object from the captured image captured by the imaging unit. And an object recognizing device configured as described above.

  According to said structure, an object recognition apparatus can recognize an object from the picked-up image which image | photographed the predetermined area | region. Therefore, it is possible to monitor an object existing in a predetermined area after specifying the type.

  The monitoring system according to the present invention preferably further includes equipment control means for controlling the operation of equipment provided in a predetermined area based on the object recognized by the object recognition device. Here, examples of equipment include production equipment at a production site, an alarm for issuing a warning, an automatic door for entering or leaving a predetermined area, and the like. In the above case, the operation of the equipment according to the object recognized by the object recognition device can be performed.

  In particular, when the above equipment is a production equipment and the object to be recognized is a worker, the production equipment is stopped only when the worker is recognized, and is operated in other cases, thereby improving the productivity of the production equipment. Can be maintained. Further, since the worker existing in the predetermined area can be surely recognized, the safety of the worker can be ensured.

  The monitoring system according to the present invention further includes an entry detection sensor that detects entry of an object into the predetermined area, and the object recognition device starts operation when the entry detection sensor detects entry of the object. It is preferable to do. Here, as an example of the approach detection sensor, there is a sensor that detects the presence of an object by emitting a wave such as an ultrasonic wave, a radio wave, and an infrared ray in the vicinity of a boundary of a predetermined region and shielding or reflecting the wave by the object. Can be mentioned. In the above case, since the object recognition apparatus is stopped until a specific object enters the predetermined area, power consumption can be suppressed.

  In order to solve the above problems, an object recognition method according to the present invention is an object recognition method for recognizing an object from a captured image including an image of one or a plurality of objects, and an image acquisition step for acquiring the captured image; A feature point extracting step for extracting feature points from the acquired captured image; a spatial position calculating step for calculating the spatial position of the extracted feature points; and object recognition for recognizing the object based on the calculated spatial position of the feature points. The object recognition step includes a clustering step for performing clustering based on the calculated spatial position of the feature points, and an object corresponding to the cluster based on the size of the cluster generated by the clustering. And an object determination step for determining.

  According to the above method, a captured image is acquired, feature points are extracted from the acquired captured image, a spatial position of the extracted feature points is calculated, and clustering is performed based on the calculated spatial position of the feature points. Thereby, a cluster similar to the shape of the object is generated. Since the object type can be specified from the object size, the object type can be specified from the cluster size.

  Each unit in the object recognition apparatus can be executed on a computer by an object recognition program. Furthermore, the object recognition program can be executed on an arbitrary computer by storing the object recognition program in a computer-readable recording medium.

  As described above, the object recognition device according to the present invention acquires a captured image, extracts a feature point from the acquired captured image, calculates a spatial position of the extracted feature point, and calculates the spatial position of the calculated feature point. By performing clustering based on this, a cluster similar to the shape of the object is generated, so that the type of the object can be specified from the size of the generated cluster.

[Embodiment 1]
An embodiment of the present invention will be described with reference to FIGS. FIG. 2 shows an overview of the monitoring system 10 in the present embodiment. As shown in the figure, the monitoring system 10 of this embodiment is applied to a production site where production is performed using an automated machine (production equipment P). At this production site, a workpiece W having a predetermined shape is conveyed to the production facility P by the conveyor C.

  As shown in FIG. 2, the monitoring system 10 includes a stereo camera (imaging device, imaging means) 11 that captures an image of the dangerous area D around the production facility P, and a captured image captured by the stereo camera 11. Based on this, it is configured to include a monitoring device (object recognition device) 12 that detects the presence of the worker M and the intrusion detection in the dangerous area D by recognizing the worker M in the dangerous area D.

  FIG. 3 shows a specific configuration within the stereo camera 11 and the monitoring device 12 in the monitoring system 10. As shown in the figure, the stereo camera 11 includes a plurality of cameras that capture a dangerous area. In the present embodiment, the stereo camera 11 includes two cameras, which are referred to as “first image input device 13” and “second image input device 14”, respectively. Each of the first and second image input devices 13 and 14 transmits the captured image signal acquired by the imaging to the monitoring device 12.

  In the present embodiment, a grayscale image having only luminance information is used as a captured image, but a color image having luminance information and color information can also be used. The first and second image input devices 13 and 14 are preliminarily calibrated with each other for various parameters necessary for the above photographing.

  Further, as shown in FIG. 3, the monitoring device 12 is configured by a PC (Personal Computer) -based computer. That is, the monitoring device 12 includes a CPU (central processing unit) 20 that executes instructions of a control program that realizes various functions, a ROM (read only memory) 21 that stores the program, and a RAM (random access memory) that expands the program. ) 22, a memory 23 for storing the program and various data, a receiving unit 24 for receiving various information from the outside, and an output unit 25 for outputting various information to the outside. The CPU 20, ROM 21, RAM 22, memory 23, and output unit 25 are connected to each other via a bus 26.

  In the present embodiment, the memory 23 includes a first input image memory 30 that stores data of a first input image captured by the first image input device 13 and data of a second input image captured by the second image input device 14. A second input image memory 31 for storing image data, an image storage memory 32 for storing data of various images, a memory 33 for storing feature point coordinates, and a cluster coordinate storage memory (storage means) 34 for storing cluster coordinates. Is included.

  On the other hand, the receiving unit 24 includes A / D conversion circuits 35 and 36 that respectively acquire photographed image signals from the first and second image input devices 13 and 14 and perform A / D (Analog to Digital) conversion. . The A / D conversion circuits 35 and 36 transmit the converted captured image data to the first and second input image memories 30 and 31 as first and second input image data, respectively.

  FIG. 1 shows a functional configuration of the monitoring device 12. As illustrated, the monitoring device 12 includes a control unit 40, a first input image acquisition unit (image acquisition unit) 41, a second input image acquisition unit (image acquisition unit) 42, and an output unit (equipment control unit) 25. It is the composition provided.

  The control unit 40 controls the operation of various configurations of the monitoring device 12 in an integrated manner, and includes a CPU 20, a ROM 21, a RAM 22, and a memory 23. Details of the control unit 40 will be described later.

  The first input image acquisition unit 41 acquires and stores a photographed image captured by the first image input device 13 of the stereo camera 11 as a first input image. The A / D conversion circuit 35 and the first input image The memory 30 is provided. Similarly, the second input image acquisition unit 42 acquires and stores a captured image captured by the second image input device 14 of the stereo camera 11 as a second input image. The second input image acquisition unit 42 stores the A / D conversion circuit 36 and the second input image. The configuration includes a two-input image memory 31.

  The output unit 25 outputs various information to the outside as described above. As an example of a specific configuration of the output unit 25, a display that performs display output, a speaker that performs audio output, an alarm device that issues an alarm, a printer that performs print output, a transmission device that transmits an output signal to the outside, and the operation of a production facility And a control device for controlling operation stop.

  Furthermore, the output unit 25 includes a work output unit 43 that outputs a work output signal generated by the control unit 40, and a worker output unit 44 that outputs a worker output signal generated by the control unit 40. It is a configuration.

  Next, details of the control unit 40 will be described. As shown in FIG. 1, the control unit 40 includes a 3D (three-dimensional) information calculation unit (feature point extraction unit, spatial position calculation unit) 50, an object recognition unit (object recognition unit) 51, and an output signal generation unit 52. It is the structure provided with.

  The 3D information calculation unit 50 calculates a spatial position of a feature point of an object (object) using the first input image and the second input image acquired by the first and second input image acquisition units 41 and 42. It is. The 3D information calculation unit 50 transmits 3D information indicating the calculated spatial position to the object recognition unit 51.

  The object recognition unit 51 acquires 3D information of all feature points from the 3D information calculation unit 50, and recognizes an object based on the acquired 3D information of all feature points. Specifically, the object recognizing unit 51 includes a clustering unit (clustering unit) 54 and a size determining unit (object determining unit) 55.

  The clustering unit 54 performs clustering processing of feature points based on the 3D information of all feature points acquired from the 3D information calculation unit 50. Further, the clustering unit 54 calculates 3D size information SOj such as height and width for each cluster j generated by the clustering process and stores it in the cluster coordinate storage memory 34.

  The size determination unit 55 compares the 3D size information SOj of each cluster j calculated by the clustering unit 54 with a predetermined determination criterion, and specifies the type of object corresponding to the cluster j. More specifically, when the difference between the 3D size information SOj of each cluster j and the size determination threshold THi of the object i is within a predetermined value TH, that is, | SOj−THi | ≦ TH. The object type corresponding to the cluster j is determined to be the object type (work or worker) indicated by the size determination threshold THi. Thereby, the type of the object can be specified from the size of the cluster similar to the shape of the object.

  Furthermore, in the present embodiment, the clustering unit 54 performs clustering processing by reducing the cluster size for the feature points constituting the cluster for which the size determination unit 55 cannot determine the object. As described above, in the present embodiment, by repeating the clustering and size determination while changing the cluster size, the type of object corresponding to each cluster can be reliably specified.

  The output signal generation unit 52 generates an output signal corresponding to the object recognized by the object recognition unit 51. More specifically, when the size determination unit 55 determines that the cluster corresponds to the workpiece W, the output signal generation unit 52 includes a workpiece output signal generation unit 56 that generates a workpiece output signal, and the cluster is an operator. When the size determination unit 55 determines that it corresponds to M, it includes a worker output signal generation unit 57 that generates a worker output signal.

  The work output signal generation unit 56 transmits the generated work output signal to the work output unit 43 of the output unit 25. On the other hand, the worker output signal generation unit 57 transmits the generated worker output signal to the worker output unit 44 of the output unit 25.

  For example, a signal instructing to continue the operation of the production facility P is output as the workpiece output signal, while a signal instructing to stop the operation of the production facility P is output as the worker output signal. In this case, since the operation of the production facility P is not stopped even when the workpiece W is carried in, productivity is maintained. Further, since the operation of the production facility P is stopped when the worker M exists, the safety of the worker is ensured.

  Next, processing operations in the control unit 40 of the monitoring device 12 having the above-described configuration will be described with reference to FIGS. FIG. 4 shows the flow of processing operations in the 3D information calculation unit 50 of the control unit 40.

  As illustrated in FIG. 4, the 3D information calculation unit 50 includes a first input image memory 30 of the first input image acquisition unit 41 and a second input image memory 31 of the second input image acquisition unit 42. When the input image data and the second input image data are read out, feature points are automatically extracted by an edge extraction operator such as a Harris operator in a preset object appearance region (step S10; hereinafter, simply “S10”). As well as the other steps). Note that the algorithm for extracting the feature points is not limited to this, and a known method such as a Sobel operator or a Laplacian operator can also be used.

  Next, the 3D information calculation unit 50 associates the feature points of the first input image with the feature points of the second input image (S11). This association can be performed by scanning the epipolar line and using a general association method such as normalized correlation.

  Next, the 3D information calculation unit 50 calculates 3D information indicating the spatial position of the feature points from the parallax between the associated feature points (S12). This 3D information is converted from a CCD central coordinate system based on an imaging device (CCD), which is a coordinate system of captured images (first and second input images), to a camera coordinate system based on two cameras. Further, the information is converted into the world coordinate system to become information indicating the actual spatial position.

  FIGS. 5A to 5C show a CCD central coordinate system, a camera coordinate system, and a world coordinate system, respectively. The CCD central coordinate system shown in FIG. 6A can be converted into the camera coordinate system shown in FIG. 5B by using the following equation (1).

  Similarly, the camera coordinate system shown in FIG. 5B can be converted into the world coordinate system shown in FIG. 5C by using the following equation (2).

  Then, the 3D information calculation unit 50 stores the calculated 3D information of each feature point in the feature point coordinate storage memory 33.

  FIG. 6 shows the flow of processing operations in the control unit 40. FIG. 7 shows the meanings of the various parameters shown in FIG. As shown in the figure, there are two types of objects to be recognized in the present embodiment: a work W having an object ID (i) of 1 and a worker M having an object ID of 2. Further, the size of the workpiece W is larger than the size of the worker M. For this reason, in the present embodiment, clustering is first performed with the size of the workpiece W to determine whether the cluster is the workpiece W. If the cluster is not the workpiece W, clustering is performed with the size of the worker M. Whether the cluster is the worker M or not is determined.

  As shown in FIG. 6, when the object recognition unit 51 reads 3D information of each feature point from the feature point coordinate storage memory 33, the object recognition unit 51 initializes the object ID (i) and the cluster ID (j) (S20). .

  Next, the clustering unit 54 calculates the distance between feature points such as the Euclidean distance for all feature points as described in Patent Document 3 (S21), and the clustering threshold value of the workpiece W which is the larger clustering threshold value. Clustering processing using TH1 is performed (S22). Specifically, a plurality of feature points whose distance between feature points is equal to or less than the clustering threshold TH1 are assumed to be feature points extracted from one object and integrated into the same cluster.

  Next, the clustering unit 50 stores the number N of clusters created by the clustering process and the attributes of each cluster in the cluster coordinate storage memory 34 (S23), and a 3D size such as the height and width of each cluster. Information (SOj) (where 1 ≦ j ≦ N and N is an integer) is calculated and stored in the cluster coordinate storage memory 34 (S24).

  Next, the size determination unit 55 reads the 3D size information (SOj) of each cluster (S25), and the difference between the 3D size information (SOj) and the size determination threshold (= clustering threshold) (THi) is within a predetermined range (TH It is determined whether it is less than () (S26).

  If the difference is less than the predetermined range (YES in S26), it is determined that the object type of the cluster is the object type (work W or worker M) indicated by the clustering threshold (THi). As a result, the object is recognized. Then, the size determination unit 55 determines whether or not the determined object type is a worker (S27). If it is determined that the object is a worker (YES in S27), the output signal for the worker is determined. The generation unit 57 generates an output signal indicating the worker M and others and outputs the output signal to the outside via the worker output unit 44 (S28). Thereafter, the processing operation is terminated.

  On the other hand, when it is determined that the user is not an operator (NO in S27), it is determined whether or not the cluster ID (j) is N or more (S29). If cluster ID (j) is smaller than N (NO in S29), the cluster to be checked remains, so after incrementing cluster ID (j) (S30), the process returns to step S24 to perform the above processing operation. repeat. On the other hand, when cluster ID (j) is N or more (YES in S29), there is no cluster to be checked, so all clusters correspond to work W. Therefore, the work output signal generation unit 56 generates an output signal indicating the work W and outputs the output signal to the outside via the work output unit 43 (S31). Thereafter, the processing operation is terminated.

  Returning to step S26, if the difference between the 3D size information (SOj) of the cluster and the size determination threshold (= clustering threshold) (THi) is greater than or equal to a predetermined range (TH) (NO in S26), the cluster Determines that the object does not correspond to the clustering threshold (THi), and cancels the cluster (S32).

  Next, the clustering unit 54 increments the object ID (i) and changes the object to be clustered (S33). Next, the clustering unit 54 determines whether or not the incremented object ID (i) is equal to or less than the number M of object types (S34). If the number is greater than the number M of object types (NO in S34), the object recognition unit 51 determines that there is an object other than the target object, and the worker output signal generation unit 57 An output signal indicating the operator M and others is generated and output to the outside via the worker output unit 44 (S28). Thereafter, the processing operation is terminated.

  On the other hand, when object ID (i) is equal to or less than the number M of object types (YES in S34), object recognition unit 51 has an object to be clustered, and therefore a clustering threshold for object ID (i). Clustering (partial clustering) using THi is performed (S35). Then, it returns to step S24 and repeats the said processing operation.

  FIG. 8 shows a clustering process performed by the clustering unit 54. As shown in the figure, first, clustering by the longest distance method is performed (S40). Next, the age of the cluster is updated (S41), and a distance matrix between the clusters is created (S42). Then, the clusters are integrated (S43), and various lists are updated (S44). Thereafter, the clustering process is terminated.

  FIG. 9 shows the clustering process performed by the clustering unit 54 in more detail. In the figure, i indicates the number of feature points, and n indicates the number of feature points. First, initial setting is performed (S50). Next, the following processing is performed for a certain feature point, and this is performed for all feature points (S51).

  That is, first, the minimum distance and the corresponding cluster number are obtained for the corresponding feature point and the previous cluster combination (S52). Next, it is determined whether or not the minimum distance is smaller than the threshold value (S53). If the minimum distance is equal to or larger than the threshold value (NO in S53), the process moves to the next feature point (S57).

  On the other hand, if the minimum distance is smaller than the threshold value (YES in S53), it is determined whether the corresponding feature point is not yet associated with the cluster (S54). If not associated (YES in S54), the cluster number and the feature point number are linked to associate the corresponding feature point with the cluster (S55). On the other hand, if it is associated (NO in S54), the distance from the cluster is set to the minimum value (S56). After the process of step S55 or step S56, the process moves to the next feature point (S57).

  FIGS. 10A to 10F specifically show processing performed by the monitoring device 12 when one work A and a worker M exist in the dangerous area D. FIG. As shown in FIG. 5A, first, the first and second input image acquisition units 41 and 42 acquire captured images.

  Next, as illustrated in FIG. 10B, the 3D information calculation unit 50 extracts feature points from the acquired captured image and calculates 3D information. Next, as shown in FIG. 5C, the clustering unit 54 performs clustering with the size of the workpiece A. As a result, two clusters are generated.

  Next, as shown in FIG. 10D, the clustering unit 54 acquires the 3D size (width, height, and depth) of each cluster, and the size determination unit 55 determines whether each cluster is a work. Whether or not is determined based on a criterion.

  For example, as shown in FIG. 10 (d), cluster 1 has a height of 175 cm, a width of 50 cm, and a depth of 50 cm, and cluster 2 has a height of 170 cm, a width of 170 cm, and a depth of 170 cm. Suppose that The workpiece judgment criteria are 165 cm to 175 cm in height, 165 cm to 175 cm in width, and 165 cm to 175 cm in depth. In this case, the size determination unit 55 determines that the cluster 1 is not the work A, and determines that the cluster 2 is the work A.

  Next, the clustering unit 54 performs partial clustering. That is, as illustrated in FIG. 10E, the cluster is canceled for the feature point of the cluster 1 determined not to be the work A, and the clustering is performed again with the size of the worker M. As a result, a new cluster 3 is generated.

  Next, as illustrated in FIG. 10F, the clustering unit 54 acquires the 3D size (width, height, and depth) of the cluster 3, and the size determination unit 55 determines that the cluster 3 is the worker M. Whether or not there is is determined based on a determination criterion.

  For example, as shown in FIG. 10F, the cluster 3 has a height of 175 cm, a width of 50 cm, and a depth of 50 cm. The determination criterion for the worker M is that the height is 165 cm to 175 cm, the width is 45 cm to 55 cm, and the depth is 45 cm to 55 cm. In this case, the size determination unit 55 determines that the cluster 3 is the worker M.

Then, when the worker M is included in the determination result of the size determination unit 55, the worker output signal generation unit 57 generates and outputs a danger signal as the worker output signal. On the other hand, when the worker M is not included in the determination result, the work output signal generation unit 56 generates and outputs a safety confirmation signal as the work output signal.
[Embodiment 2]
Next, another embodiment of the present invention will be described with reference to FIGS. The monitoring system of the present embodiment is different from the monitoring system 10 shown in FIGS. 1 to 10 in that the worker M is larger in size than the work W, and the other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 11 shows the meanings of the various parameters shown in FIG. As shown in the figure, there are two types of objects to be recognized in the present embodiment: an operator M whose object ID (i) is 1 and a work W whose object ID is 2. In addition, since the size of the worker M is larger than the size of the workpiece W, in the present embodiment, clustering is first performed with the size of the worker M, and it is determined whether or not the cluster is the worker M. If it is not the worker M, clustering is performed with the size of the work W, and it is determined whether or not the cluster is the work W.

  12A to 12D specifically show processing performed by the monitoring device 12 when one work B and a worker M exist in the dangerous area D. FIG. FIGS. 13A to 13C specifically show processing performed by the monitoring device 12 when the worker M does not exist in the dangerous area D and only one work B exists. is there.

  First, as shown in FIG. 12A, the first and second input image acquisition units 41 and 42 acquire captured images. Next, as illustrated in FIG. 12B, the 3D information calculation unit 50 extracts feature points from the acquired captured image and calculates 3D information. Next, the clustering unit 54 performs clustering with the size of the worker M, as shown in FIG. As a result, two clusters are generated.

  Next, as shown in FIG. 12D, the clustering unit 54 acquires the 3D size of each cluster, and the size determination unit 55 determines whether each cluster is an operator based on the determination criterion. judge.

  For example, as shown in FIG. 12D, the cluster 1 has a height of 175 cm, a width of 50 cm, and a depth of 50 cm, and the cluster 2 has a height of 30 cm, a width of 30 cm, and a depth of 30 cm. Suppose that Then, the determination criteria of the worker are assumed to be 150 cm to 180 cm in height, 30 cm to 80 cm in width, and 30 cm to 80 cm in depth. In this case, the size determination unit 55 determines that the cluster 1 is the worker M and determines that the cluster 2 is not the worker M. Since the determination result of the size determination unit 55 includes the worker M, the worker output signal generation unit 57 generates and outputs a danger signal as the worker output signal.

  As shown in FIG. 13A, when the worker M does not exist in the danger area D and only one work B exists, the worker M is not included in the determination result, so clustering is performed. The unit 54 releases the cluster and performs clustering again with the size of the work B as shown in FIG. As a result, a new cluster 1 is generated.

  Next, as illustrated in FIG. 13C, the clustering unit 54 acquires the 3D size of the cluster 1, and the size determination unit 55 determines whether the cluster 1 is the work B based on the determination criterion. judge.

  For example, as shown in FIG. 13C, the cluster 1 has a height of 30 cm, a width of 30 cm, and a depth of 30 cm. The determination criteria for the workpiece B are 25 cm to 35 cm in height, 25 cm to 35 cm in width, and 25 cm to 35 cm in depth. In this case, the size determination unit 55 determines that the cluster 1 is the work B. As described above, since the size determination unit 55 does not include the worker M but includes the workpiece B, the workpiece output signal generation unit 56 generates and outputs a safety confirmation signal as the workpiece output signal.

Therefore, also in this embodiment, the type of object can be specified from the cluster size similar to the shape of the object, as in the above embodiment. In addition, since clustering is repeated for each object size, the type of object corresponding to each cluster can be reliably identified.
[Embodiment 3]
Next, another embodiment of the present invention will be described with reference to FIGS. The monitoring system of this embodiment differs from the monitoring system 10 shown in FIGS. 1 to 10 in the number of objects to be recognized, and the other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  14 and 15 show the flow of processing operations in the control unit 40. FIG. 16 shows the meanings of various parameters shown in FIGS. 14 and 15. As shown in the figure, the objects to be recognized in the present embodiment are the work A having the object ID (i) of 1, the worker M having the object ID of 2, and the work B having the object ID of 3. It is a kind. The size of the workpiece A is larger than the size of the worker M, and the size of the worker M is larger than the size of the workpiece B.

  For this reason, in this embodiment, clustering is first performed with the size of the work A to determine whether or not the cluster is the work A. If the cluster is not the work A, the clustering is performed with the size of the worker M. It is determined whether or not the cluster is the worker M. If the cluster is not the worker M, clustering is performed with the size of the work B to determine whether or not the cluster is the work B.

  Specifically, as shown in FIGS. 14 and 15, the clustering unit 54 first reads the 3D information of each feature point from the feature point coordinate storage memory 33, as described in Patent Document 3 above, The distance between feature points such as the Euclidean distance is calculated, and clustering processing is performed using the upper clustering threshold TH1max of the work A, which is the largest clustering threshold (S60). Specifically, a plurality of feature points whose distance between the feature points is equal to or less than the clustering threshold TH1max are assumed to be feature points extracted from one object and integrated into the same cluster.

  Next, the clustering unit 50 obtains 3D size information such as the height and width of each cluster created by the clustering process (S61). Next, for a certain cluster, the size determination unit 55 includes the 3D size information of the cluster between the upper limit value TH1max and the lower limit value TH1min (see FIG. 16) of the size determination threshold value (= clustering threshold value) of the workpiece A. It is determined whether it is within the range of the size determination threshold value of the workpiece A (S62).

  If the 3D size information of the cluster is within the size determination threshold value of work A (YES in S62), size determination unit 55 determines that the object of the cluster is work A, and the work The output signal generator 56 generates and outputs a work A output signal (S63). As a result, the object is recognized. Thereafter, the process proceeds to step S75.

  On the other hand, when the 3D size information of the cluster is outside the range of the size determination threshold value of the workpiece A (NO in S62), the size determination unit 55 further determines that the 3D size information of the cluster is the workpiece A size. It is determined whether or not the size determination threshold value is smaller than the upper limit value TH1max (S64). When the 3D size information of the cluster is equal to or larger than the upper limit value TH1max of the size determination threshold value of the work A, the size determination unit 55 determines that the cluster is an object larger than the work A, that is, the work A, the worker, The output signal generator 52 determines that the object is larger than any of the workpiece B and the workpiece B, and generates another output signal (S65). Thereafter, the process proceeds to step S75.

  On the other hand, when the 3D size information of the cluster is smaller than the upper limit value TH1max of the size determination threshold value of the workpiece A, the clustering unit 54 determines the workpiece B, which is the smallest clustering threshold value, for the feature points constituting the cluster. Using the upper limit clustering threshold TH2max, the clustering process is performed as described above (S66). Next, the clustering unit 50 obtains 3D size information such as the height and width of each cluster created by the clustering process (S67).

  Next, for a certain cluster, the size determination unit 55 includes the 3D size information of the cluster between the upper limit value TH2max and the lower limit value TH2min (see FIG. 16) of the size determination threshold value (= clustering threshold value) of the work B. It is determined whether it is within the range of the size determination threshold value of the workpiece B (S68).

  When the 3D size information of the cluster is within the range of the size determination threshold value of work B (YES in S68), size determination unit 55 determines that the object of the cluster is work B, and the work The output signal generator 56 generates and outputs a work B output signal (S69). As a result, the object is recognized. Thereafter, the process proceeds to step S75.

  On the other hand, when the 3D size information of the cluster is outside the range of the size determination threshold value for work B (NO in S68), size determination unit 55 further determines that the 3D size information of the cluster is the work B size. It is determined whether or not the size determination threshold value is larger than a lower limit value TH1min (S70). When the 3D size information of the cluster is equal to or smaller than the lower limit value TH2min of the workpiece B size determination threshold (NO in S70), the size determination unit 55 determines that the object is smaller than the workpiece B, that is, the workpiece A, work The output signal generation unit 52 determines that the object is smaller than either the worker or the work B, and generates another output signal (S65). Thereafter, the process proceeds to step S75.

  On the other hand, when the 3D size information of the cluster is larger than the lower limit value TH2min of the size determination threshold value of work B (YES in S70), the clustering unit 54 performs intermediate clustering on the feature points constituting the cluster. The clustering process is performed as described above using the upper limit clustering threshold TH3max of the worker M, which is the threshold (S71). Next, the clustering unit 50 obtains 3D size information such as the height and width of each cluster created by the clustering process (S72).

  Next, for a certain cluster, the size determination unit 55 determines that the 3D size information of the cluster is between the upper limit value TH3max and the lower limit value TH3min (see FIG. 16) of the size determination threshold (= clustering threshold) of the worker M. It is determined whether it is included, that is, whether it is within the range of the size determination threshold value of the worker M (S73).

  When the 3D size information of the cluster is within the size determination threshold value of worker M (YES in S73), size determination unit 55 determines that the object of the cluster is worker M. The worker output signal generator 57 generates and outputs a worker output signal (S74). As a result, the object is recognized. Thereafter, the process proceeds to step S75.

  On the other hand, when the 3D size information of the cluster is outside the range of the size determination threshold of worker M (NO in S73), the output signal generation unit 52 determines that the object could not be recognized, An output signal is generated (S65). Thereafter, the process proceeds to step S75.

  In step S75, it repeats until all the clusters produced | generated by step S60 regarding the workpiece | work A perform the said determination process (S62-S74). Similarly, all of the clusters generated in step S66 for the workpiece B are repeated until the above determination processing (S65, S68 to S74) is performed (S76). Furthermore, it repeats until all the clusters produced | generated by step S71 regarding the worker M perform the said determination process (S65 * S73 * S74) (S77). Thereafter, the processing operation is terminated.

  17A to 17H specifically show the processing performed by the monitoring device 12 when there is one work A, one work B, and one worker M in the dangerous area D. FIG. It is shown. As shown in FIG. 5A, first, the first and second input image acquisition units 41 and 42 acquire captured images.

  Next, as illustrated in FIG. 17B, the 3D information calculation unit 50 extracts feature points from the acquired captured image and calculates 3D information. Next, as shown in FIG. 5C, the clustering unit 54 performs clustering with the size of the workpiece A. As a result, two clusters 1 and 2 are generated.

  Next, as illustrated in FIG. 17D, the clustering unit 54 acquires the 3D sizes (width, height, and depth) of the clusters 1 and 2. Next, the size determination unit 55 determines whether or not the cluster 1 is the work A based on the determination criterion. In the case shown in the figure, the size determination unit 55 determines that the cluster 1 is not the work A, so that the clustering unit 54 applies the work to the feature points constituting the cluster 1 as shown in FIG. Clustering is performed with the size of B. As a result, two clusters 3 and 4 are generated.

  Next, the clustering unit 54 acquires the 3D size (width, height, and depth) of the clusters 3 and 4. Next, the size determination unit 55 determines whether or not the cluster 3 is the work B based on the determination criterion. In the illustrated case, the size determination unit 55 determines that the cluster 3 is the work B.

  Similarly, the size determination unit 55 determines whether or not the cluster 4 is the work B based on the determination criterion. In the illustrated case, since the size determination unit 55 determines that the cluster 4 is the work A, the clustering unit 54 performs clustering with the size of the worker M on the feature points constituting the cluster 4. Thereby, one cluster 4 'is generated. Next, the clustering unit 54 acquires the 3D size (width, height, and depth) of the cluster 4 ′, and the size determination unit 55 determines whether the cluster 4 ′ is the worker M based on the determination criterion. Judgment. In the illustrated case, the size determination unit 55 determines that the cluster 4 ′ is the worker M.

  Accordingly, the object recognizing unit 51 recognizes the cluster 3 as the work B and recognizes the cluster 4 ′ as the worker M as shown in FIG.

  Next, as shown in FIG. 17G, the size determination unit 55 determines whether or not the cluster 2 is the work A based on the determination criterion. In the illustrated case, the size determination unit 55 determines that the cluster 2 is the work A.

  As described above, the object recognition unit 51 recognizes cluster 2 as work A, recognizes cluster 3 as work B, recognizes cluster 4 ′ as worker M, as shown in FIG. The output signal generation unit 52 outputs an output signal corresponding to each object.

Therefore, in the present embodiment, first, with respect to the object of the maximum size among the objects to be specified, clustering is performed with a cluster size corresponding to the size of the object, and then the cluster reference is made on the basis of the size corresponding to the size of the object. Size determination is performed. Next, among the objects to be specified, clustering and size determination are performed in the same manner with respect to the object of the minimum size. Accordingly, when a cluster having a size that deviates from the size range of the object to be specified is acquired, an abnormality can be reported immediately.
[Embodiment 4]
Next, still another embodiment of the present invention will be described with reference to FIGS. The monitoring system of the present embodiment is different from the monitoring system 10 shown in FIGS. 1 to 10 in that a cluster location is added to a determination condition for determining whether a certain cluster is a work. The configuration of is the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  In the above embodiment, the 3D size of a cluster is used as a determination condition for determining whether or not a certain cluster is a work. However, when the 3D size of the workpiece and the worker is similar, the determination result based on the 3D size may not be sure.

  On the other hand, the workpiece usually moves in a predetermined area at the production site. For example, in the example illustrated in FIG. 2, the workpiece W is moving on the transporter C. For this reason, the position where the workpiece W exists is limited to the transporting machine C. Therefore, in the present embodiment, as the determination condition, the cluster location is used together with the 3D size of the cluster.

  FIG. 18 shows a schematic configuration of the control unit 40 in the monitoring system 10 of the present embodiment. Compared with the control unit 40 shown in FIG. 1, the control unit 40 of the present embodiment includes a position determination unit (object determination unit) 60 that determines whether or not the center of gravity position of the cluster is within a predetermined range. The difference is included in the object recognition unit 51, and the other configurations are the same. The position determination unit 60 is provided between the size determination unit 55 and the output signal generation unit 52.

  The position determination unit 60 calculates the centroid position of the cluster determined by the size determination unit 55 as a work, determines whether or not the calculated centroid position is included in an area where the work normally moves, and if it is included, A cluster is determined to be a work, and if it is not included, the cluster is determined to be other than a work (for example, an operator).

  FIG. 19 shows a processing operation added in the present embodiment to the processing operation of the control unit 40 shown in FIG. In the present embodiment, when the size determination unit 55 determines that the determined object type is not a worker (NO in S27), the position determination unit 60 selects a cluster corresponding to the determined object type. The center of gravity position is calculated (S80). Next, the position determination unit 60 determines whether or not the calculated barycentric position is included in a region where the workpiece normally moves, that is, whether or not the workpiece is within the existing range (S81).

  If within the work existence range (YES in S81), it is determined that the cluster is a work, and the process proceeds to step S29. On the other hand, if the workpiece is out of the existing range (NO in S81), it is determined that the cluster is not a workpiece but an operator, and the process proceeds to step S28.

  Therefore, in the monitoring system 10 of the present embodiment, the type of the object is determined based on the cluster size and the cluster location, so that a more reliable determination result of the object can be obtained.

  The position information may be absolute 3D information in the detection space including the dangerous area D or may be relative 3D information based on a certain object.

Further, there may be a case where the worker cannot enter the area where the workpiece normally moves. Therefore, the position determination unit 60 calculates the center of gravity position of the cluster determined by the size determination unit 55 as an operator, determines whether or not the calculated center of gravity position is included in the region where the workpiece normally moves. In some cases, it is possible to determine that the cluster is not an operator, and to determine that the cluster is an operator if the cluster is not included.
[Embodiment 5]
Next, still another embodiment of the present invention will be described with reference to FIGS. Compared to the monitoring system 10 shown in FIGS. 18 and 19, the monitoring system of this embodiment uses a moving speed of a cluster instead of a cluster location as a determination condition for determining whether a certain cluster is a work. The other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  Usually, the workpiece moves at a predetermined speed range on the production site. For example, in the example illustrated in FIG. 2, the speed of the workpiece W is the transport speed of the transport machine C. Therefore, in this embodiment, the speed of the cluster is used together with the 3D size of the cluster as the determination condition.

  FIG. 20 shows a schematic configuration of the control unit 40 in the monitoring system 10 of the present embodiment. Compared to the control unit 40 shown in FIG. 18, the control unit 40 in the present embodiment determines whether or not the moving speed of the center of gravity position of the cluster is within a predetermined range instead of the position determination unit 60. The difference is that the determination unit (object determination unit) 61 is included in the object recognition unit 51, and other configurations are the same. The speed determination unit 61 is provided between the size determination unit 55 and the output signal generation unit 52.

  Specifically, the speed determination unit 61 calculates the position of the center of gravity of the cluster determined by the size determination unit 55 as a work and stores it in the cluster coordinate storage memory 34. In addition, the speed determination unit 61 identifies the previous cluster corresponding to the cluster based on the 3D size or the like, and reads the center of gravity position of the identified previous cluster from the cluster coordinate storage memory 34.

  Next, the speed determination unit 61 calculates the movement distance of the cluster based on the calculated center of gravity position and the read center of gravity position, and uses the period between the time when the previous image was acquired and the time when the current image was acquired. To calculate the movement speed of the cluster. Then, the speed determination unit 61 determines whether or not the calculated movement speed of the cluster is included in the range of the movement speed at which the work normally moves. If included, the speed determination unit 61 determines that the cluster is a work. If not, the cluster is determined to be other than work.

  FIG. 21 shows a processing operation added in the present embodiment to the processing operation of the control unit 40 shown in FIG. As illustrated, in this embodiment, when the size determination unit 55 determines that the determined object type is not an operator (NO in S27), the speed determination unit 61 determines the determined object type. The barycentric position of the cluster corresponding to is calculated (S80). Next, the speed determination unit 61 calculates the moving distance of the cluster as described above (S82), calculates the moving speed of the cluster, and the calculated moving speed is the moving speed at which the workpiece normally moves. It is determined whether it falls within the range (S83).

  If it is within the moving speed range of the workpiece (YES in S83), it is determined that the cluster is a workpiece, and the process proceeds to step S29. On the other hand, when the workpiece is out of the moving speed range (NO in S83), it is determined that the cluster is not a workpiece but an operator, and the process proceeds to step S28.

Therefore, in the monitoring system 10 of the present embodiment, since the type of the object is determined based on the cluster size and the moving speed of the cluster, a more reliable determination result of the object can be obtained.
[Embodiment 6]
Next, still another embodiment of the present invention will be described with reference to FIG. 22 and FIG. Compared to the monitoring system 10 shown in FIGS. 18 and 19, the monitoring system according to the present embodiment uses a determination condition for determining whether a certain cluster is a work as a determination condition for determining whether a certain cluster is a work instead of the cluster location. Is different, and the other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  Usually, the workpiece is transported along a predetermined path at the production site. For example, in the example shown in FIG. 2, the traveling direction of the workpiece W is the conveyance direction of the conveyance machine C. Therefore, in this embodiment, the traveling direction of the cluster is used together with the 3D size of the cluster as the determination condition.

  FIG. 22 shows a schematic configuration of the control unit 40 in the monitoring system 10 of the present embodiment. Compared with the control unit 40 shown in FIG. 18, the control unit 40 in this embodiment determines whether or not the traveling direction of the center of gravity position of the cluster is within a predetermined range instead of the position determination unit 60. The difference is that the direction recognition unit (object determination unit) 62 is included in the object recognition unit 51, and the other configurations are the same. The traveling direction determination unit 62 is provided between the size determination unit 55 and the output signal generation unit 52.

  Specifically, the traveling direction determination unit 62 calculates the position of the center of gravity of the cluster determined by the size determination unit 55 as a work and stores it in the cluster coordinate storage memory 34. In addition, the traveling direction determination unit 62 identifies the previous cluster corresponding to the cluster based on the 3D size or the like, and reads the center of gravity position of the identified previous cluster from the cluster coordinate storage memory 34.

  Next, the traveling direction determination unit 62 calculates the traveling direction of the cluster based on the calculated center of gravity position and the read center of gravity position. Then, the traveling direction determination unit 62 determines whether or not the calculated traveling direction of the cluster is included within a range of the traveling direction in which the workpiece normally moves. If included, the traveling direction determination unit 62 determines that the cluster is a workpiece. If not, the cluster is determined to be other than work.

  FIG. 23 shows a processing operation added in the present embodiment to the processing operation of the control unit 40 shown in FIG. As illustrated, in this embodiment, when the size determination unit 55 determines that the determined object type is not an operator (NO in S27), the traveling direction determination unit 62 determines the determined object type. The center-of-gravity position of the cluster corresponding to the type is calculated (S80). Next, the traveling direction determination unit 62 calculates the traveling direction of the cluster as described above (S84), and determines whether or not the calculated traveling direction is included in the traveling direction range in which the workpiece normally moves. Judgment is made (S85).

  If within the range of the workpiece traveling direction (YES in S85), it is determined that the cluster is a workpiece, and the process proceeds to step S29. On the other hand, if the workpiece is outside the range of the moving direction of the work (NO in S85), it is determined that the cluster is not a work but an operator, and the process proceeds to step S28.

Therefore, in the monitoring system 10 according to the present embodiment, the type of the object is determined based on the cluster size and the traveling direction of the cluster, so that a more reliable determination result of the object can be obtained.
[Embodiment 7]
Next, still another embodiment of the present invention will be described with reference to FIGS. Compared with the monitoring system 10 shown in FIG. 18 and FIG. 19, the monitoring system of this embodiment uses the number of clusters instead of the cluster location as a determination condition for determining whether a certain cluster is a work. The point of use is different, and the other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  Usually, a workpiece is limited in its production capacity on the transfer capacity (the number per unit time or the number per unit area) of a transfer machine that transfers the workpiece. For example, in the example shown in FIG. 2, the number of workpieces W in the dangerous area D is four. Therefore, in the present embodiment, the number of clusters is used together with the 3D size of the clusters as the determination condition.

  FIG. 24 shows a schematic configuration of the control unit 40 in the monitoring system 10 of the present embodiment. Compared to the control unit 40 shown in FIG. 18, the control unit 40 in this embodiment is a number determination unit (object that determines whether the number of clusters is within a predetermined range, instead of the position determination unit 60. The determination unit 63 is included in the object recognition unit 51, and the other configurations are the same. The number determination unit 63 is provided between the size determination unit 55 and the output signal generation unit 52.

  Specifically, the number determination unit 63 calculates the position of the center of gravity of the cluster determined by the size determination unit 55 as a work, and calculates the number of the clusters. Next, the number determination unit 63 determines whether or not the calculated number is included in the range of the number of workpieces that normally exist in the dangerous area D. If included, the number determination unit 63 determines the cluster as a workpiece, If not included, the cluster is determined to be other than work.

  FIG. 25 shows a processing operation added in the present embodiment to the processing operation of the control unit 40 shown in FIG. As illustrated, in this embodiment, when the size determination unit 55 determines that the determined object type is not a worker (NO in S27), the number determination unit 63, as described above, The number of clusters corresponding to the determined object type is calculated (S86). Next, the number determination unit 63 determines whether or not the calculated number is included in the range of the number of workpieces that normally exist in the dangerous area D (specified value) (S87).

  If the number is within the range (YES in S87), it is determined that the cluster is a work, and the process proceeds to step S29. On the other hand, if the number is out of the above range (NO in S87), it is determined that the workpiece is not in a predetermined number, and the process proceeds to step S28 as in the case where the worker M is detected.

Therefore, in the monitoring system 10 of the present embodiment, the type of the object is determined based on the cluster size and the number of clusters, so that a more reliable determination result of the object can be obtained.
[Embodiment 8]
Next, still another embodiment of the present invention will be described with reference to FIGS. 26 and 27. FIG. The monitoring system 10 according to the present embodiment determines whether a certain cluster is a work by performing statistical processing by combining the determination conditions described in the above-described embodiments. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 26 shows a schematic configuration of the control unit 40 in the monitoring system 10 of the present embodiment. The control unit 40 in the present embodiment includes a combination determination unit (object determination unit) 64 and a statistical determination processing unit (object determination unit) 65 in the object recognition unit 51, as compared with the control unit 40 shown in FIG. In other respects, other configurations are the same. The combination determination unit 64 and the statistical determination processing unit 65 are sequentially provided between the size determination unit 55 and the output signal generation unit 52.

  The combination determining unit 64 includes a position determining unit 60 shown in FIG. 18, a speed determining unit 61 shown in FIG. 20, a traveling direction determining unit 62 shown in FIG. 22, and a number determining unit 63 shown in FIG. . That is, the combination determination unit 64 determines whether or not the cluster is a workpiece from the cluster existing position, moving speed, traveling direction, and number of clusters that the size determination unit 55 has determined as a workpiece. is there. The combination determination unit 64 notifies each determination result to the statistical determination processing unit 65.

The statistical determination processing unit 65 determines whether the cluster is a work by performing a statistical process on each determination result of the combination determination unit 64. Specifically, the statistical determination processing unit 65 performs the above determination by calculating using the evaluation function of the following equation (3).
f = αs × fs + αp × fp + αv × fv + αd × fd + αn × fn (3)
Here, fx indicates each determination result. Αx is a weighting coefficient of each determination result, and expresses the determination reliability of each determination process. The subscripts s, p, v, d, and n indicate size determination, existence position determination, speed determination, traveling direction determination, and number determination, respectively. Note that the determination reliability can be set by a system designer or the like.

  FIG. 27 illustrates a processing operation added in the present embodiment to the processing operation of the control unit 40 illustrated in FIG. 6. As illustrated, in the present embodiment, when the size determination unit 55 determines that the determined object type is not a worker (NO in S27), the combination determination unit 64 determines that the type of the object is as described above. The center-of-gravity position of the cluster corresponding to the determined object type is calculated (S80). Next, the combination determination unit 64 performs the presence position determination process S81 as shown in FIG. 19, the movement speed determination processes S82 and S83 as shown in FIG. 21, and the traveling direction as shown in FIG. The determination processes S84 and S85 and the existence number determination processes S86 and S87 as shown in FIG. 25 are performed to obtain the respective determination results.

  Next, the statistical determination processing unit 65 calculates by assigning each determination result acquired by the combination determination unit 64 to the evaluation function of the above equation (3), and uses the calculation result as a final determination result (S88). ). If it is determined that the cluster is a work (YES in S88), the process proceeds to step S29. On the other hand, if it is determined that the cluster is not a work (NO in S88), it is determined that the cluster is an operator, and the process proceeds to step S28.

Therefore, since the monitoring system 10 of the present embodiment uses various determination results, it is possible to determine whether a cluster is a work even in a wide variety of environments. That is, more robust determination is possible.
[Embodiment 9]
Next, still another embodiment of the present invention will be described with reference to FIGS. Compared to the monitoring system 10 shown in FIG. 18 and FIG. 19, the monitoring system 10 of this embodiment uses a 2D of a cluster instead of a cluster location as a determination condition for determining whether a certain cluster is a work. The difference is that (two-dimensional) information is added, and the other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  In the above-described embodiment, it is assumed that a feature serving as a reference for determining a workpiece exists in the 3D information, but there may be a case where the feature does not exist. In this case, it is difficult to determine the workpiece in the above embodiment. Therefore, in the present embodiment, a 2D mark that can identify the type of object is applied in advance to a predetermined position on the surface of the workpiece, and the 2D mark is extracted from the 2D image and recognized to recognize the workpiece. Can be determined.

  FIG. 28 shows a schematic configuration of the control unit 40 in the monitoring system 10 of the present embodiment. Compared with the control unit 40 shown in FIG. 18, the control unit 40 in this embodiment includes a 2D information determination unit (object determination unit) 66 that determines a workpiece from 2D image information instead of the position determination unit 60. The points included in the recognition unit 51 are different, and the other configurations are the same. The 2D information determination unit 66 is provided between the size determination unit 55 and the output signal generation unit 52.

  Specifically, the 2D information determination unit 66 calculates the center-of-gravity position of the cluster for which the size determination unit 55 could not identify the type of object. Next, the 2D information determination unit 66 has the calculated cluster center-of-gravity position in a predetermined area in the image (in the example shown in FIG. 2, the area where the workpiece is transferred on the transfer machine C), and the image It is determined whether or not the 2D mark is included in the above-described cluster. If the 2D mark is included, the cluster is determined as a work, and if not included, the cluster is determined as a non-work.

  Since the image may be a 2D image, it may be either the first input image acquired by the first input image acquisition unit 41 or the second input image acquired by the second input image acquisition unit 42.

  FIG. 29 shows a processing operation added in the present embodiment to the processing operation of the control unit 40 shown in FIG. As illustrated, in the present embodiment, when the size determination unit 55 determines that the determined object type is not a worker (NO in S27), the 2D information determination unit 66 determines the determined object type. The center-of-gravity position of the cluster corresponding to the type is calculated (S80).

  Next, the 2D information determination unit 66 determines whether or not the calculated center of gravity position exists in the reading position of the 2D mark, that is, in the region where the workpiece to which the 2D mark is applied moves (S90). If the workpiece does not exist in the moving area (NO in S90), it is determined that the cluster is not a workpiece but an operator, and the process proceeds to step S28.

  On the other hand, if the workpiece exists in the moving area (YES in S90), the 2D information determination unit 66 extracts a 2D mark from the area in the 2D image (S91) and recognizes it (S92). The recognition of the 2D mark can be realized by using a general template matching technique.

  Then, the 2D information determination unit 66 determines whether or not the recognized 2D mark is a specified mark (S93). If it is a specified mark (YES in S93), the cluster is a work. The process proceeds to step S29. On the other hand, if the mark is not a prescribed mark (NO in S93), it is determined that the cluster is not a work but an operator, and the process proceeds to step S28.

Therefore, the monitoring system 10 according to the present embodiment can determine a workpiece from 2D information even when there is no characteristic information serving as a criterion for workpiece determination in the 3D information.
[Embodiment 10]
Next, still another embodiment of the present invention will be described with reference to FIGS. The monitoring system 10 of this embodiment is different from the monitoring system 10 shown in FIGS. 1 to 10 in that the cluster is tracked, and the other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 30 shows a schematic configuration of the control unit 40 in the monitoring system 10 of the present embodiment. The control unit 40 in the present embodiment is different from the control unit 40 shown in FIG. 1 in that the object recognition unit 51 includes a tracking processing unit (estimating unit, cluster integrating unit) 67 that performs cluster tracking. The configuration of is the same. The tracking processing unit 67 is in communication with the clustering unit 54.

  After the clustering unit 54 performs the above-described clustering process, the tracking processing unit 67 stores 3D information and cluster IDs related to one or all clusters in the cluster coordinate storage memory 34 as cluster history information. In addition, the tracking processing unit 67 calculates the movement amount of the cluster using the cluster history information in the plurality of past frames stored in the cluster coordinate storage memory 34, and thereby determines the location of the cluster in the current frame. presume. Then, the tracking processing unit 67 integrates the feature points of the current frame that appear at the estimated location of the cluster before the clustering unit 54 performs the clustering process.

  FIG. 31 shows the flow of processing operations of the control unit 40 in the present embodiment. The processing operation of the control unit 40 in this embodiment is different from the processing operation of the control unit 40 shown in FIG. 6 in that a processing operation in the tracking processing unit 67 is added, and the other processing operations are the same. .

  As shown in FIG. 31, the tracking processing unit 67 reads the 3D information from the cluster coordinate storage memory 34 and calculates the movement amount for each cluster ID (S94). Next, the tracking processing unit 67 estimates the appearance position of the cluster in the next frame (S95). Next, the tracking processing unit 67 reads out from the feature point coordinate storage memory 33 and extracts the feature points to be clustered for the cluster at the estimated appearance position (S96). Next, the tracking processing unit 67 integrates the extracted feature points in advance as a tracking cluster (S97), and notifies the clustering unit 54 of the 3D information of the tracking cluster and the information of the feature points constituting the tracking cluster.

  Therefore, in the monitoring system 10 of this embodiment, since the current cluster location is estimated from the past movement history of the cluster, the cluster can be specified with high accuracy, and the type of object corresponding to the cluster can be specified with high accuracy. In addition, since the feature points appearing at the estimated positions of the clusters are integrated before the clustering unit 54 performs the clustering process, operations performed in the clustering process such as the calculation of the Euclidean distance can be reduced. The processing speed can be increased.

  32 and 33 specifically show the tracking process. As shown in FIG. 32A, first, the first and second input image acquisition units 41 and 42 acquire a captured image at time t.

  Next, the 3D information calculation unit 50 extracts feature points from the acquired captured image to calculate 3D information, and as shown in FIG. 32B, the clustering unit 54 is in the clustering target region. Clustering is performed on feature points. The clustering target area at this time is a normal wide area. As a result of the clustering, a cluster as shown in FIG. 7C is generated, and the 3D information and the barycentric position of the generated cluster are stored in the cluster coordinate storage memory 34.

  Next, as shown in FIG. 33A, the first and second input image acquisition units 41 and 42 acquire a captured image at time t + 1. Next, the 3D information calculation unit 50 extracts feature points from the acquired captured image and calculates 3D information.

  Next, the tracking processing unit 67 reads the cluster 3D information and the barycentric position at time t from the cluster coordinate storage memory 34, and estimates the appearance area of the cluster at time t + 1. Thereby, as shown in FIG. 33B, the clustering target region can be limited.

  Then, the clustering unit 54 performs clustering in the clustering target region to generate a cluster as shown in FIG. 33C, and the 3D information and the centroid position of the generated cluster are stored in the cluster coordinate storage memory 34. Remembered.

Using the cluster centroid position at time t and the cluster centroid position at time t + 1 stored in the cluster coordinate storage memory 34, the movement amount of the cluster can be calculated. Therefore, the tracking processor 67 can estimate the appearance area of the next cluster with high accuracy by using the calculated movement amount.
[Embodiment 11]
Next, still another embodiment of the present invention will be described with reference to FIG. The monitoring system 10 of the present embodiment is different from the monitoring system 10 shown in FIGS. 1 to 10 in that an active sensor is used instead of a stereo camera, and the other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted. The active sensor calculates 3D information of an object by irradiating the object with a light beam having a predetermined pattern, shooting the object, and acquiring a 2D captured image of the object.

  FIG. 34 shows a schematic configuration of the monitoring system 10 of the present embodiment. The monitoring system 10 of this embodiment uses an active sensor (imaging means) 70 instead of the stereo camera 11 as compared with the monitoring system 10 shown in FIG.

  The active sensor 70 includes a projector 71, an image input device 72, and a 3D information calculation unit 73. The projector 71 irradiates an object with a light beam having a predetermined pattern.

  The image input device 72 captures an object irradiated with a predetermined pattern and acquires a 2D captured image of the object. The image input device 72 transmits the acquired 2D captured image information to the 3D information calculation unit 73.

  The 3D information calculation unit 73 calculates 3D information of the object using the 2D captured image acquired from the image input device 72. The 3D information calculation unit 73 transmits the calculated 3D information to the clustering unit 54. The configuration downstream from the clustering unit 54 is the same as the configuration shown in FIG.

Accordingly, the active sensor 70 can be used instead of the stereo camera 11 as in the monitoring system 10 of the present embodiment. In this case, since the active sensor 70 calculates the 3D information of the object, the monitoring device 12 includes the first input image acquisition unit 41, the second input image acquisition unit 42, and the 3D information calculation unit 50 as illustrated in FIG. Can be omitted.
[Embodiment 12]
Next, still another embodiment of the present invention will be described with reference to FIGS. Compared with the monitoring system 10 shown in FIGS. 1 to 10, the monitoring system 10 according to the present embodiment adds a detection signal from another sensor to a determination condition for determining whether a certain cluster is a work. Differently, other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 35 shows an overview of the monitoring system 10 in the present embodiment. The monitoring system 10 of the present embodiment is different from the monitoring system 10 shown in FIG. 2 in that an infrared camera (object detection sensor) 75 is added, and other configurations are the same. The infrared camera 75 is installed so as to overlook an area where the stereo camera 11 captures an image, that is, an object detection area. The infrared camera 75 images the detection area and transmits information of the captured infrared image to the monitoring device 12.

  FIG. 36 shows a schematic configuration of a main part of the monitoring device 12. Compared with the monitoring device 12 shown in FIG. 1, the monitoring device 12 of the present embodiment further includes an infrared image acquisition unit (detection signal acquisition unit) 76, and the other sensor determination unit 77 (object determination unit) is an object. The points included in the recognition unit 51 are different, and the other configurations are the same. The other sensor determination unit 77 is provided between the size determination unit 55 and the output signal generation unit 52.

  The infrared image acquisition unit 76 acquires an infrared image from the infrared camera 75. The infrared image acquisition unit 76 transmits information on the acquired infrared image to the other sensor determination unit 77.

  The other sensor determination unit 77 calculates the temperature of each cluster using the infrared image received from the infrared image acquisition unit 76, and determines the type of the object based on the calculated temperature. For example, if the temperature of a certain cluster is around 36 ° C., the other sensor determination unit 77 can determine that the cluster is an operator.

  FIG. 37 shows a processing operation added in the present embodiment to the processing operation of the control unit 40 shown in FIG. In the present embodiment, when the size determination unit 55 determines that the determined object type is not a worker (NO in S27), the other sensor determination unit 77 selects a cluster corresponding to the determined object type. Is calculated (S80).

  Next, the other sensor determination part 77 calculates | requires the measurement temperature of each cluster by extracting the area | region in the gravity center position of each cluster from the infrared image which the infrared image acquisition part 76 acquired (S100). Next, it is determined whether or not the measured temperature of each cluster is within a predetermined temperature range (body temperature range) (S101).

  If it is within the predetermined temperature range (YES in S101), it is determined that the cluster is an operator, and the process proceeds to step S28. On the other hand, if the temperature is outside the predetermined temperature range (NO in S101), it is determined that the cluster is a work, and the process proceeds to step S29.

Therefore, in the monitoring system 10 of the present embodiment, since the type of the object is determined based on the cluster size and the measured temperature of the cluster, a more reliable determination result of the object can be obtained.
[Embodiment 13]
Next, still another embodiment of the present invention will be described with reference to FIG. The monitoring system 10 of the present embodiment is different from the monitoring system 10 shown in FIGS. 1 to 10 in that the monitoring operation is started when an object enters the detection area, and the other configurations are the same. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 38 shows an overview of the monitoring system 10 in the present embodiment. The monitoring system 10 of the present embodiment is different from the monitoring system 10 shown in FIG. 2 in that an entry detection sensor 80 that detects that an object has entered the detection area is provided at the boundary of the detection area. The configuration is the same.

  As the approach detection sensor 80, a sensor that detects the presence of an object by emitting a wave such as an ultrasonic wave, a radio wave, or an infrared ray in the vicinity of the boundary and shielding or reflecting the wave by the object is used. The entry detection sensor 80 notifies the monitoring device 12 when an entry of an object is detected. Thereby, the monitoring apparatus 12 starts the monitoring operation by starting the photographing of the stereo camera 11 and starting the operation of various components in the monitoring apparatus 12.

Therefore, in the monitoring system 10 according to the present embodiment, when the object does not exist in the detection area, the monitoring operation is stopped, so that useless processing operations can be avoided.
[Embodiment 14]
Next, still another embodiment of the present invention will be described with reference to FIG. FIG. 39 shows an overview of the monitoring system 10 in the present embodiment. The monitoring system 10 of the present embodiment is different from the monitoring system 10 shown in FIG. 2 in that a mat sensor (object detection sensor) 81 for detecting the presence of an object is provided, and other configurations are the same. is there. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

When the stereo camera (imaging device) 11 performs shooting, as shown in FIG. 39, the back side of the object a becomes a blind spot, and the object b existing on the back side of the object a cannot be recognized. Therefore, in the present embodiment, when a blind spot area exists in the captured image from the stereo camera 11, the monitoring device 12 uses the object detection result by the mat sensor 81 together with the recognition of the object using the captured image. is doing. Thereby, it is possible to ensure the safety by recognizing the object even in a blind spot in the shooting by the stereo camera 11.
[Embodiment 15]
Next, another embodiment of the present invention will be described with reference to FIG. The monitoring system 10 of this embodiment shows an application example of the monitoring system 10 shown in FIGS. In addition, the same code | symbol is attached | subjected to the structure similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 40 shows an overview of the monitoring system 10 in the present embodiment. Compared to the monitoring system 10 shown in FIG. 2, the monitoring system 10 of the present embodiment is provided with a wall around the detection region, and a part of the wall is opened and an automatic door 82 is provided.

  The monitoring device 12 controls the automatic door 82 to operate when an operator (human) is recognized near the automatic door 82 and stops the operation of the automatic door 82 in other cases. Thereby, even if a non-human work or a small animal exists near the automatic door 82, the automatic door 82 can be prevented from opening and closing.

  The monitoring device 12 can also be controlled to operate the automatic door 82 when recognizing a workpiece near the automatic door 82 and to stop the operation of the automatic door 82 in other cases. In this case, what comes out of the automatic door 82 can be limited to only workpieces. This is suitable when the work is transported from the production site to a room where it is not preferable for a human or a small animal to enter, such as a sterilization room.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  Finally, each block of the calibration device 13, particularly the control unit 16, may be configured by hardware logic, or may be realized by software using a CPU as follows.

  That is, the calibration device 13 includes a CPU (central processing unit) that executes instructions of a control program for realizing each function, a ROM (read only memory) that stores the program, a RAM (random access memory) that expands the program, A storage device (recording medium) such as a memory for storing the program and various data is provided. An object of the present invention is to provide a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program of the calibration device 13 which is software that realizes the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying the calibration device 13 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the calibration device 13 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention can be applied not only to a cluster by creating a cluster from feature points of a photographed image but also to specify a spatial position of the object, so that it can be applied to a monitoring system that tracks an object.

It is a block diagram which shows schematic structure of the monitoring apparatus in the monitoring system which is one Embodiment of this invention. It is a perspective view which shows the outline | summary of the said monitoring system. It is a block diagram which shows the specific structure in the stereo camera and monitoring apparatus in a monitoring system. It is a flowchart which shows the flow of the processing operation in the 3D information calculation part of the control part of the said monitoring apparatus. It is a figure which shows the coordinate system utilized in the said 3D information calculation part, The figure (a) shows the CCD center coordinate system, The figure (b) shows the camera coordinate system, The figure (c) ) Indicates the world coordinate system. It is a flowchart which shows the flow of the processing operation in the said control part. It is a figure which shows the meaning of the various parameters shown by FIG. 6 in a table format. It is a flowchart which shows the flow of the processing operation in the clustering part of the said control part. It is a flowchart which shows the flow of the processing operation in the said clustering part in detail. FIGS. 4A to 4F are diagrams specifically showing processing performed by the monitoring apparatus when one work and one worker are present in the dangerous area. In the monitoring system which is another embodiment of this invention, it is a figure which shows the meaning of the various parameters shown by FIG. 6 in a table format. FIGS. 9A to 9D are diagrams specifically showing processing performed by the monitoring device of the monitoring system when one work and one worker are present in the dangerous area. FIGS. 9A to 9C are diagrams specifically showing processing performed by the monitoring apparatus when only one workpiece exists in the dangerous area. It is a flowchart which shows the flow of the processing operation in the control part of the monitoring apparatus of the monitoring system which is another embodiment of this invention. It is a flowchart which shows the flow of the processing operation in the said control part. It is a figure which shows the meaning of the various parameters shown by FIG. 14 and FIG. 15 in a table format. FIGS. 11A to 11H are diagrams specifically showing processing performed by the monitoring device 12 when two works and one worker are present in the dangerous area. It is a block diagram which shows schematic structure of the control part in the monitoring apparatus of the monitoring system which is further another embodiment of this invention. It is a flowchart which shows the processing operation of the said control part, and has shown the processing operation added to the processing operation shown by FIG. It is a block diagram which shows schematic structure of the control part in the monitoring apparatus of the monitoring system which is further another embodiment of this invention. It is a flowchart which shows the processing operation of the said control part, and has shown the processing operation added to the processing operation shown by FIG. It is a block diagram which shows schematic structure of the control part in the monitoring apparatus of the monitoring system which is further another embodiment of this invention. It is a flowchart which shows the processing operation of the said control part, and has shown the processing operation added to the processing operation shown by FIG. It is a block diagram which shows schematic structure of the control part in the monitoring apparatus of the monitoring system which is further another embodiment of this invention. It is a flowchart which shows the processing operation of the said control part, and has shown the processing operation added to the processing operation shown by FIG. It is a block diagram which shows schematic structure of the control part in the monitoring apparatus of the monitoring system which is further another embodiment of this invention. It is a flowchart which shows the processing operation of the said control part, and has shown the processing operation added to the processing operation shown by FIG. It is a block diagram which shows schematic structure of the control part in the monitoring apparatus of the monitoring system which is further another embodiment of this invention. It is a flowchart which shows the processing operation of the said control part, and has shown the processing operation added to the processing operation shown by FIG. It is a block diagram which shows schematic structure of the control part in the monitoring apparatus of the monitoring system which is further another embodiment of this invention. It is a flowchart which shows the processing operation of the said control part. FIGS. 9A to 9C are diagrams specifically showing processing performed by the tracking processing unit in the control unit. FIGS. 9A to 9C are diagrams specifically showing processing performed by the tracking processing unit. It is a block diagram which shows schematic structure of the monitoring system which is another embodiment of this invention. It is a perspective view which shows the outline | summary of the monitoring system which is another embodiment of this invention. It is a block diagram which shows schematic structure of the principal part of the monitoring apparatus in the said monitoring system. It is a flowchart which shows the processing operation of the control part of the said monitoring apparatus, and has shown the processing operation added to the processing operation shown by FIG. It is a perspective view which shows the outline | summary of the monitoring system which is another embodiment of this invention. It is a perspective view which shows the outline | summary of the monitoring system which is another embodiment of this invention. It is a perspective view which shows the outline | summary of the monitoring system which is other embodiment of this invention.

Explanation of symbols

10 surveillance system 11 stereo camera (photographing means)
12 Monitoring device (object recognition device)
25 Output unit (equipment control means)
34 Cluster coordinate storage memory (storage means)
41 1st input image acquisition part (image acquisition means)
42 Second input image acquisition unit (image acquisition means)
50 3D information calculation unit (feature point extraction means, spatial position calculation means)
51 Object recognition unit (object recognition means)
54 Clustering unit (clustering means)
55 Size determination unit (object determination means)
60 Position determination part (object determination means)
61 Speed determination unit (object determination means)
62 Traveling direction determination unit (object determination means)
63 Number determination unit (object determination means)
64 combination determination unit (object determination means)
65 Statistical judgment processing part (object judgment means)
66 2D information determination unit (object determination means)
67 Tracking processing unit (estimating means, cluster integrating means)
70 Active sensor (photographing means)
75 Infrared camera (object detection sensor)
76 Infrared image acquisition unit (detection signal acquisition means)
77 Other sensor determination unit (object determination means)
80 Entry detection sensor 81 Mat sensor (object detection sensor)

Claims (16)

An object recognition apparatus for recognizing an object from a captured image including an image of one or more objects,
Image acquisition means for acquiring a captured image;
Feature point extraction means for extracting feature points from the acquired captured image;
Spatial position calculation means for calculating the spatial position of the extracted feature points;
Object recognition means for recognizing the object based on the calculated spatial position of the feature point;
The object recognition means includes clustering means for performing clustering based on the calculated spatial positions of the feature points;
An object recognition apparatus comprising: an object determination unit that determines an object corresponding to a cluster based on a size of the cluster generated by the clustering. The clustering means performs clustering with a size corresponding to a certain object,
The object determination means determines whether the cluster corresponds to the object;
2. The object according to claim 1, wherein the clustering unit performs clustering again with a size corresponding to another object with respect to a feature point constituting a cluster that is determined not to correspond to the object by the object determination unit. Recognition device. The clustering means includes
Perform clustering at the maximum size among multiple sizes corresponding to multiple objects to be recognized,
The object recognition apparatus according to claim 2, wherein clustering is performed again at a minimum size among the plurality of sizes with respect to feature points constituting a cluster that is determined not to correspond to the object by the object determination unit. The image acquisition means acquires two captured images from a stereo camera,
The object recognition apparatus according to claim 1, wherein the spatial position calculation unit calculates a spatial position of the feature point using a feature point in the two captured images. The image acquisition means acquires a captured image from an active sensor,
The spatial position calculation unit acquires spatial position information regarding the captured image from the active sensor, and calculates the spatial position of the feature point based on the acquired spatial position information. The object recognition apparatus described in 1.   The object determination means determines an object corresponding to the cluster based on the size of the cluster generated by the clustering and at least one of the existence position, the traveling direction, the moving speed, and the number of the cluster. The object recognition apparatus according to claim 1.   The object determination means performs statistical processing on the object determination based on the size of the cluster and the object determination based on at least one of the existence position, the traveling direction, the moving speed, and the number related to the cluster. The object recognition apparatus according to claim 6, wherein an object corresponding to the cluster is determined. The surface of the object is provided with a mark that can identify the object,
The object determination unit extracts the mark from the captured image region corresponding to the spatial position of the cluster generated by the clustering, and corresponds to the cluster based on the extracted mark and the size of the cluster. The object recognition apparatus according to claim 1, wherein an object is determined. It further comprises detection signal acquisition means for acquiring a detection signal from a sensor that detects an object,
The said object determination means determines the object corresponding to the said cluster based on the determination of the object by the size of the said cluster, and the detection of the object by the said detection signal acquisition means. Object recognition device. Storage means for storing the history information of the spatial position of the cluster generated by the clustering;
Estimating means for estimating a spatial region in which the cluster is next located based on history information of the spatial position of the cluster;
A cluster integration unit that extracts feature points included in the spatial region of the cluster estimated by the estimation unit out of the feature points extracted by the feature point extraction unit, and integrates the feature points into the cluster; The object recognition apparatus according to claim 1. A monitoring system for monitoring a predetermined area,
Photographing means for photographing the predetermined area;
A monitoring system comprising: the object recognizing device according to any one of claims 1 to 10, wherein an object is recognized from a photographed image photographed by the photographing means.   The monitoring system according to claim 11, further comprising equipment control means for controlling operation of equipment provided in a predetermined area based on an object recognized by the object recognition device. It further comprises an entry detection sensor for detecting the entry of an object into the predetermined area,
The monitoring system according to claim 11, wherein the object recognition device starts an operation when the entry detection sensor detects the entry of the object. An object recognition method for recognizing an object from a captured image including an image of one or more objects,
An image acquisition step of acquiring a captured image;
A feature point extraction step of extracting feature points from the acquired captured image;
A spatial position calculating step for calculating a spatial position of the extracted feature points;
An object recognition step for recognizing the object based on the calculated spatial position of the feature point,
The object recognition step includes a clustering step for performing clustering based on the calculated spatial position of the feature points;
And an object determination step of determining an object corresponding to the cluster based on the size of the cluster generated by the clustering.   An object recognition program for operating the object recognition apparatus according to any one of claims 1 to 10, wherein the object recognition program causes a computer to function as each of the means.   A computer-readable recording medium on which the object recognition program according to claim 15 is recorded.

Images