JP5350444B2 - Method and apparatus for automatic visual event detection - Google Patents

Abstract

Disclosed are methods and apparatus for automatic visual detection of events, for recording images of those events and retrieving them for display and human or automated analysis, and for sending synchronized signals to external equipment when events are detected. An event corresponds to a specific condition, among some time-varying conditions within the field of view of an imaging device, that can be detected by visual means based on capturing and analyzing digital images of a two-dimensional field of view in which the event may occur. Events may correspond to rare, short duration mechanical failures for which obtaining images for analysis is desirable. Events are detected by considering evidence obtained from an analysis of multiple images of the field of view, during which time moving mechanical components can be seen from multiple viewing perspectives.

Description

[Background of the invention]
The present invention relates to high-speed video event detection, motion analysis, image recording, and automatic image analysis.

[Description of related technology]
It is well known in the art to use high-speed image recording devices for analytical operations in mechanical systems that are too fast for the human eye to see. These devices display images as slow motion or still images and acquire hundreds or thousands of images per second of machine processing to view and analyze high-speed machine operations for human users , To record.

  Of particular interest is the recording of rare, short-term mechanical events that can cause failures in machine processing. The fact that events are rare and short-lived creates special challenges. For example, if the image recording device records 1000 images per second, the event lasts 3 milliseconds and occurs once per average hour. Without additional equipment, an average user would need to view 3.6 million photos to find 2 or 3 events including events.

  It is well known in the art to address this challenge by providing an image recording device trigger signal to indicate when an event occurs. The image recorder saves a limited number of the latest images, and if the trigger signal indicates that an event has occurred, it says it is the last second of the recording, records a simple additional time, and exits . This gives the user a small number of each image to see both before and after the event. Furthermore, the user knows reliably if each image was acquired in relation to the time of the event indicated by the trigger signal.

  Obviously, the success of this method depends on whether an accurate trigger signal can be generated. The use of photodetectors for this purpose is well known in the art. An exemplary photodetector has a light source and a single optoelectronic sensor that correspond to the intensity of light reflected by a point on the surface of the object or transmitted along a path through which the object may pass. The user adjustable sensitivity threshold establishes the upper (or lower) light intensity that applies a voltage to the output signal of the photodetector.

  That is often the case when multiple photodetectors need to provide a trigger signal. For example, if the machine process is a production discrete object on the production line, the event corresponds to the production of the object due to the lack of parts, one to detect when the object is present, and another to detect the part shortage To do so, at least two photodetectors are required. Sometimes more than one detector is needed to detect complex events.

However, the use of the photodetector for supplying the trigger signal has several drawbacks as follows.
A simple measurement of the intensity of light transmitted or reflected at one or more points may not be sufficient to detect an event.
• It is difficult to adjust the position of each photodetector so that you can see exactly the points.
• Measured points should not be moved around during normal operation of mechanical processing.
• The need for multiple photodetectors makes installation and configuration difficult.

  The use of machine vision systems to provide trigger signals is well known in the art. A machine vision system is a device that can acquire a digital image of a two-dimensional field of view and analyze and determine the image. The image is acquired by exposing the two-dimensional arrangement of the photosensitive member to light focused on the arrangement by the lens in a short period of time called integration time or shutter time. The arrangement is called an imaging device and the individual elements are called pixels. Each pixel measures the intensity of light going towards it at the shutter time. The measured intensity value is converted to a digital number and stored in the memory of a vision system for imaging, which is a digital processing element such as a computer using methods well known in the art for determination. Is analyzed.

  Machine vision systems can avoid the limitations of photodetectors. A single machine vision system can be replaced with many photodetectors, making advanced measurements of a wide range of brightness patterns instead of single point intensity measurements. The viewing position can be adjusted using a graphical user interface instead of a screwdriver or wrench, and their position can be moved based on the contents of its own image .

However, the machine vision system has the following inherent disadvantages.
Machine vision systems are generally only suitable when events are related to the inspection of individual objects.
• Machine vision systems cannot detect events that are too slow and last only for a short time, so we must look for long-lasting conditions caused by events such as defective products.

  Note that when the machine vision system is used to provide a trigger signal, it is separate from the high speed image recording device. It is not possible to see images containing detected events and to analyze images acquired from the device. Even if those images are available in a machine vision system, they are incredibly high and are produced and are not analyzed by a machine vision system of a conventional design.

  A visual detector method and apparatus are novel methods and systems that can overcome the above-described limitations of prior art photodetectors and machine vision systems for detecting when a trigger event has occurred. teach. These teachings also provide materials for inventions that lead to improvements within the scope of the original teachings. The following section briefly summarizes the method and apparatus of the visual detector, and the following sections present the problems addressed by the present invention.

[Visual detector method and apparatus]
A visual detector method and apparatus is based on acquiring a digital image of a two-dimensional field of view where an object to be detected and inspected can be located, analyzing and determining the image, and automatic optoelectronic detection of the object. And supply systems and methods for inspection. These systems and methods analyze brightness patterns reflected from a wide area, address many of the distinguishable features of the object, provide line switching through software means, and place uncertain available object locations. Correspond. They are less expensive than prior art machine vision systems, are easier to set up and can be operated at much faster speeds. These systems and methods also take advantage of the many perspectives of moving objects, operate without triggers, provide appropriate synchronized output signals, and other important and useful features that will be apparent to those skilled in the art. Supply ability.

  One aspect of a visual detector method and apparatus is an apparatus, called a visual detector, that can acquire and analyze a series of images faster than prior art visual systems. Such a series of images acquired and analyzed is called a frame. If what is moving passes through the field of view (FOV), the rate at which a frame is acquired and analyzed, called the frame rate, is said to be high enough that it is seen in a number of consecutive frames. Since the object is moving between successive frames, it is placed in many positions in the field of view, and is therefore seen from a number of visual viewpoints and positions associated with illumination.

  Another aspect of the visual detector method and apparatus is called dynamic image analysis, where the object is in the field of view and multiple frames are acquired and analyzed, resulting in a composite of evidence obtained from each of the frames. Is a method for inspecting an object based on The method provides significant advantages over prior art opportunity visual systems that determine based on a single frame.

  Further, another aspect of the visual detector method and apparatus is a method for detecting an event that may occur in the field of view, called visual event detection. An event is an object that passes through the field of view and uses visual event detection to detect the object without the need for a trigger signal.

  Additional aspects of the visual detector method and apparatus will become apparent in the study and detailed description of the figures presented therein.

  In order to obtain images from multiple viewpoints, it is preferred that the object to be detected and inspected move at least a few pixels, at least a small fraction of the field of view, between successive frames. According to the method and apparatus of the visual detector, it is generally preferred that the motion of the object be no more than about 1/4 of the FOV per frame, and in an exemplary embodiment is only 5% or less of the FOV. It is preferable. This is preferably achieved not only by slowing down the manufacturing process, but also by providing a sufficiently high frame rate. In the example system, the frame rate is at least 200 frames per second, and in another example, the frame rate is at least 40 times the average rate at which objects appear in the visual detector.

  The exemplary system is taught to acquire and analyze up to 500 frames per second. This system utilizes an ultra-sensitive imaging device with much fewer pixels than prior art vision systems. High sensitivity allows for very short shutter times using low cost LEDs, which in turn allows very short acquisition times with the synthesis of a small number of pixels. The imaging device connects to a digital signal processor (DSP) that can receive and store pixel data dynamically in analytical operations. The time to analyze each frame is taught within the time required to acquire the next frame, generally using the method implemented using the adapted software for the DSP and executed. . Acquisition and analysis methods and devices are combined to provide the preferred high frame rate. Using the objects of the present invention and carefully matching the capabilities of the imaging device, DSP and illumination, the exemplary system can be significantly less expensive than prior art machine vision systems.

  The method of visual event detection involves taking successive frames and analyzing each frame to determine the evidence that an event has occurred or has occurred. If visual event detection is used to detect an object without the need for a trigger signal, the analysis determines evidence that the object is located in the field of view.

  In the exemplary method, the evidence is in the form of a value that indicates the level of confidence that the object is located in the field of view, referred to as object detection weight. The value may be a range of confidence levels, or a number indicating some item of information leading to evidence, simply a yes / no choice indicating high or low confidence. One example of such a number is the so-called fuzzy logic value further described there. It should be noted that the machine cannot make a perfect decision from the image and will make a judgment based on incomplete evidence.

  When performing object detection, each frame is tested to determine if there is sufficient evidence that the object is in the field of view. If simply a yes / no value is used, the evidence can be considered sufficient if the value is “yes”. If numbers are used, sufficiency may be determined by comparing numbers up to a threshold. A frame with sufficient evidence is called a valid frame. It should be noted here that, based on an understanding of the specific application at hand, what is ultimately defined by the human user setting up the visual detector is sufficient evidence to be composed. . The visual detector adapts to the definition to make that decision automatically.

  When performing object detection, each object that passes through the field of view generates a number of useful frames due to the high frame rate of the visual detector. However, if an object passes through the field of view, which may be in several viewpoints or other states, these frames are not exactly continuous because there is not enough evidence that the event is in the field of view. There is. Therefore, when a valid frame is found, it is preferable not to end the object detection until many consecutive invalid frames are found. This number is appropriately selected by the user.

  Once it is found that the set of valid frames can correspond to an object passing through the field of view, it is preferable to perform further analysis to determine whether the object was actually detected. This further analysis can take into account valid frame statistics, including the number of valid frames, the sum of object detection weights, average object detection weights, and the like.

  The method of dynamic image analysis includes acquiring and analyzing multiple frames for inspecting an object, and “inspecting” means determining information about the state of the object. In one example of this method, the condition of the object includes whether the object meets the inspection criteria selected as needed by the user.

  In an aspect of the visual detector method and apparatus, the effective frame selected by the visual event detection method is that used by the dynamic image analysis method to inspect the object, so dynamic image analysis is visual. Combined with event detection. In other aspects of the visual detector method and apparatus, the frame used by dynamic image analysis can be obtained in response to a trigger signal.

  Each such frame is analyzed to determine evidence that the object meets the inspection criteria. In one exemplary method, in the form of a value that indicates the level of confidence that the object meets the inspection criteria, referred to as the object pass score. Like object detection weights, its value simply indicates a number such as a high or low confidence, a range of confidence levels, or a fuzzy logic value indicating some item of information leading to evidence, Yes / no choice.

  The state of the object may be determined from an object pass score statistic, such as an average or percentage of the object pass score. The state may also be determined by weighted statistics, such as weighted average or weighted percentage, using object determination weights. Weighted statistics weight as more important evidence from the frame, characterized by its confidence being high that the object is actually located in the field of view of the frame.

  Evidence for object detection and inspection is obtained by inspecting a frame for information about one or more visible features of the object. A visible feature is a portion of an object where the amount, pattern, or other feature of the emitted light conveys information about the appearance, uniqueness, and state of the object. The light may be reflected, transmitted, or refracted from a source outside or inside the object, or may be a source inside the object, but not limited to some process or combination of processes. Radiated.

  One aspect of the method and operation of the visual detector includes object detection weights and object pass scores, whereas the evidence is obtained by image analysis operations in one or more regions of interest in each frame where evidence is required. Is the method. In one example of this method, the image analysis operation calculates a measurement based on pixel values in the region of interest, and the measurement is responsive to an appropriate feature of the visible feature of the object. The measurements are converted to logical values by threshold manipulation, and the logical values obtained from the region of interest are combined to form frame evidence. A logical value is a binary or fuzzy logical value that uses a binary or fuzzy threshold and logical combination as required.

  For visual event detection, evidence that the object is located in the field of view is effectively defined by the region of interest, measurement, threshold, logic synthesis, and other parameters further described herein, which are collectively This is called the vision detector setting and is chosen by the user as needed for a given application of the present invention. Similarly, visual detector settings are defined as constituting sufficient evidence.

  For dynamic image analysis, evidence that an object meets the inspection criteria is also effectively defined by the settings of the visual detector.

[Examination of problems]
The given limits of photodetectors and machine vision systems that provide triggers for fast event detection, motion analysis, and image recording integrate it with fast image recording by providing fast visual event detection Thus, there is a need for improved methods and systems that avoid the need for trigger signals.

  The visual detector method and apparatus do not learn integration with image recording for use in motion analysis, but with other advantages, new image analysis methods and systems that provide fast visual event detection. Thus, using preferred image recording and display capabilities to achieve a new and useful method and system for automatic visual detection, recording and retrieval of events, a method combining preferred elements and visual detector methods and device settings. And the system needs to be improved.

  In addition, the visual detector methods and apparatus provide illustrative examples of visual event detection primarily for detecting events corresponding to individual objects passing through the field of view. It will be apparent to those skilled in the art that these professors can be used to detect other event types, but improvements not taught therein can also help detect such events. Therefore, there is a need to broaden the visual event detection professor to improve its utility for detecting various events.

[Summary of Invention]
The present invention provides a method and system for automatic visual detection, recording and retrieval of events. In this specification:
“Event” refers to a specific condition among some time-varying conditions within the field of view of the imaging device that can be detected by visual means.
“Automatic visual detection” means that an event is detected based on the content of the image acquired by the imaging device without requiring human intervention or an external trigger signal.
“Recording” means the corresponding image before, during and / or after the event is stored in memory.
“Search” means that these images are searched for purposes such as display for human users and further automatically analyzed by an image analysis system.

  The methods and systems disclosed herein may be used for any purpose such as, but not limited to, sending an external device that an event has occurred, and providing a synchronized output pulse that indicates when the event has occurred. Therefore, it is useful for automatic visual detection of events. It is more useful for high speed motion analysis of mechanical processing, and any other application for short rare event imaging is desirable.

  In accordance with the disclosure of the present invention, a vision detector or other suitable device may have a time-varying condition that has a field of view such as a mechanical process in which an event corresponding to some special condition may occur. And configured to detect an event. The vision detector acquires a series of frames, each frame being an image of the field of view. Further, using any of a variety of methods and systems, such as, but not limited to, those disclosed in the methods and apparatus of visual detectors, events in the field of view can be seen in more detail below. The frame is analyzed to obtain evidence that it has occurred.

  When an event occurs, the analysis identifies a set of “event frames” that reveal sufficient evidence that the event has occurred. A set may contain only one event frame, but may also contain multiple event frames. In the exemplary embodiment disclosed herein, an event frame is continuous in a series of frames, but an embodiment may be devised within the scope of the present invention even if the event frames are not completely continuous. Is easy.

  Consider the following example: The detected event corresponds to a kinematic machine component moving out of tolerance. The vision detector is configured to detect an error range component, which is an area of the field of view outside the acceptable range. In some machine cycles, it is assumed that the component moves through the error range for three consecutive frames. In addition, analysis of the frame shows strong evidence that the component is in error range in the first and third frames, and weak evidence that the component is in error range in the second frame. I assume that it becomes clear. This may occur because the viewpoint or position of the component relative to the illumination of the second frame is a viewpoint or position where the component is difficult to see. The analysis also reveals that the component is unlikely to have many frames in its range before and after three important frames.

  The set of event frames begins when the first frame reveals evidence that the event has occurred, and subsequent frames show evidence that the event continues Finished in the third frame when there is no. The positive evidence that combines the first and third frames and the weak evidence of the second frame are considered sufficient to conclude that the event has occurred. In this example, three frames are event frames.

  In some embodiments, the second frame is not considered an event frame. The choice of whether to consider the second frame as an event frame can be considered either within the scope of the present invention. In the exemplary embodiment disclosed herein, an event frame is considered continuous and includes a second frame.

  If the evidence is determined to be sufficient to determine that an event has occurred, a plurality of selected frames are selected from a series of frames and recorded in memory. The frame is selected for recording depending on the event frame or the position in the series of frames relative to the recording time calculated as described below. The event frame itself may be recorded, but the frame before the event frame in the series of frames may be recorded. In addition, a frame after an event frame in a series of frames may be recorded. In an exemplary embodiment, frames acquired within a time interval specified by the user for the recording time are recorded. In another exemplary embodiment, a predetermined number of frames are recorded, such as an event frame and consecutive frames immediately before and immediately after the event frame.

  Frames from these stored selection frames are retrieved in response to commands, display for human users using a graphical user interface to issue commands, and even issue commands Used for various purposes, such as further automated image analysis by existing image analysis systems.

  In embodiments where frames are displayed for a human user, it is generally desirable to display several frames at a time, but typically enough for the user to see useful details in each image. It is impractical to display all images recorded at a different display resolution at once. In accordance with the present invention, a portion of the recorded frame is displayed at once. The user selects a displayed part by issuing a scroll command for moving the displayed part back and forth. The portion displayed at a time preferably includes several frames, but can include at least one frame.

  In an exemplary embodiment, the frame is displayed using a graphical user interface (GUI). The frame portion displayed at once is included in the “projection slide window” of the GUI, and the frame portion is displayed as a continuous “thumbnail” image with low resolution. The resolution of the thumbnail images is chosen to be low enough to see a useful number of images at once, or high enough to see the details of each image. The scroll command is provided by a conventional GUI element.

  This exemplary embodiment further displays one frame of the frame portion at full resolution in an image display window. When the projection slide is moved back and forth by the scroll command, the frame displayed in the image display window is also moved back and forth.

  In an exemplary embodiment, evidence that an event has occurred in the field of view is obtained in each frame in the form of a value called “event detection weight” that indicates the level of confidence that the event has occurred or has occurred. The value may be a simple yes / no selection indicating a high or low trust level, a number indicating a range of trust levels, or any item of information conveying evidence. An example of such a number is called a fuzzy logic value and will be described in detail later. There is no machine that can make perfect decisions from images, so decisions are based on incomplete evidence.

  Event detection weights are obtained for each frame by image analysis operations on one or more target regions of the frame. In an exemplary embodiment, each image analysis operation calculates a measurement based on the pixel value of the region of interest, where the measurement corresponds to the amount of light, pattern, or other feature of the region. The measurement is converted to a logical value by a threshold operation, and the logical value obtained from the target region is combined to calculate the event detection weight of the frame. The logical value is a binary or fuzzy logical value, and the combination of threshold and logic is binary or fuzzy as required.

In the exemplary embodiment, the event frame in the following cases is a set of consecutive frames.
-The event detection weight of the first and last frame of the set exceeds the threshold.
-After the last frame, there are at least some determined number of frames whose event detection weight does not exceed the threshold.
-The event detection weight of the frame set satisfies some determined conditions.

For this purpose, any suitable conditions may be defined, but the conditions are limited to:
The number of frames in the set,
・ Average event detection weight,
-Event detection weighted sum,
Median event detection weight,
The number of frames or the percentage of frames in the set whose event detection weight exceeds the threshold.

  The frame recorded according to the present invention is preferably stamped at the acquired time. In order to allow detailed investigation of the event, it is most useful, for example, to time stamp relative to the time of occurrence of the event itself, rather than the time of day. An event frame defines the overall time range of an event, but not a specific time. To obtain a specific effective time of an event, a mark type may be used as disclosed in the visual detector method and apparatus. As disclosed, the recording time is the time at which the object crosses some imaginary fixed reference point and can be accurately calculated according to the disclosure.

  However, not all events correspond to objects that cross a reference point. For example, an event may correspond to a stroke motion in which a machine component moves back and forth in the field of view. In such a case, it is usually more useful to define the recording time as a stroke vertex rather than an intersection of reference points. The movement around the reference point is called a flow event, and the previous and subsequent movements are called stroke events. The present invention includes a method and system for selecting either a flow or stroke event, and a method and system for calculating the recording time of a stroke event (the recording time for a flow event is determined by a visual detector). Disclosed in the method and apparatus).

  Event detection according to the present invention is an example of visual event detection disclosed in the method and apparatus of a visual detector. “The method of visual event detection determines the acquisition of successive frames and the evidence that an event has occurred or has occurred. To be involved in the analysis of each frame for ". As disclosed herein, visual event detection is primarily for embodiments in which it is generally preferred to inspect an object, with the detected event corresponding to a separate object passing through the field of view. Directed. The reader will notice that there is a great similarity between the object detection weight and object passage score disclosed herein and the event detection weight used in the present invention. Indeed, any method or apparatus disclosed herein for obtaining object detection weights or object passage scores can be used to obtain event detection weights. The only difference is the way in which these weights and scores are used, not how they are obtained.

  Furthermore, detecting that an object has passed through the field of view, or more specifically, detecting that a bad object has passed through the field of view, may be desirable for detecting, recording and retrieving events in accordance with the present invention. It is an example of an event. Certainly, there is little difference between detecting an object passing through the field of view and detecting a machine component that falls within the error range as in the above example. Thus, the disclosure of a visual detector method and apparatus can generally be considered an exemplary embodiment of the present invention, and a recording and retrieval method and system, and an improved event detection method and system are described herein. Added by disclosure of the certificate.

  The present invention will become apparent upon reading the detailed description in conjunction with the accompanying drawings.

1 illustrates an illustrative embodiment of a system for detection, recording and retrieval of visual events according to the present invention. In this case, the event corresponds to the application of a defective label of an object moving on the production line. 2 shows a timeline illustrating a typical operating cycle for a system for detecting events according to the present invention. 6 shows a flowchart illustrating an analysis procedure performed by an exemplary embodiment. In an exemplary embodiment, it is shown how evidence is weighted for visual event detection. Fig. 4 illustrates data collected and used for event detection in an exemplary embodiment. 1 shows a schematic diagram of a system according to the invention. FIG. 2 shows a block diagram of an illustrative embodiment of a vision detector that can be used as part of a system according to the present invention. In an exemplary embodiment, fuzzy logic elements used to weight and determine evidence, such as determining whether an object is present and whether to pass an inspection, are shown. FIG. 4 illustrates an organization of a set of software elements (eg, computer readable medium program instructions) used to analyze, determine, sense input, and control output signals, according to an exemplary embodiment. FIG. 6 illustrates an HMI portion for user settings of event detection parameters used to further describe an exemplary embodiment of visual event detection. FIG. FIG. 5 shows a portion of an example vision detector setup that may be used to detect a label that is incorrectly applied to an example object. FIG. 12 shows another part of the setting corresponding to the setup of the example of FIG. Fig. 4 illustrates a portion of an example vision detector setup that may be used to detect an example object in the wrong position relative to a label application arm. 14 shows another part of the setting corresponding to the setup example of FIG. FIG. 6 illustrates a portion of an example vision detector configuration that may be used to detect under-stretched or over-stretched label application arms in a manufacturing cycle. FIG. 16 illustrates how evidence is weighted to detect under-extended or over-extended arms in the example setup of FIG. 1 illustrates one way to set up the present invention to detect events corresponding to continuous web defects. FIG. 4 shows a timing diagram used to describe how the output signal is synchronized to the time of occurrence of an event. It shows how the recording time is calculated in the case of a stroke event. An HMI part for user setting of an output signal is shown. Fig. 4 shows a memory array for recording an image of detected events. FIG. 2 shows a portion of a graphical user interface that includes a projected slide window and an image display window and displays images recorded and retrieved in response to an event.

[Basic operation of the present invention]
FIG. 1 illustrates an exemplary embodiment of a vision detector configured to detect certain events that may occur on a production line and to record and retrieve an image of the events. The conveyor 100 conveys objects such as the object examples 110, 112, 114, 116 and 118, and from the left to the right, a labeling machine 160 which operates to apply a label to each object, such as the label 120 of the object 112. pass. The labeling machine 160 includes a motion arm 162 that operates to apply each label to each object. The labeling machine 160 and arm 162 are shown for purposes of illustrating the present invention and do not necessarily represent any particular machine used for labeling in industrial production.

  The labeling machine 160 sometimes malfunctions, and for example, the label 122 incorrectly attached is generated so that the lower right corner is bent on the surface of the object 116. Many industrial production lines use some form of automatic photoelectric detection to detect and reject defective products 116, such as photodetectors, machine vision systems, or vision detectors. Such inspection is valuable in preventing defective products from being delivered to consumers, but in the first place does not help prevent defective products from being manufactured. One object of the present invention is to assist a manufacturing engineer in diagnosing and correcting the cause of a defective product by providing an image showing that the defective product has actually been created.

  In addition to transporting objects for manufacturing purposes, the conveyor 100 generates relative motion between the objects and the vision detector 130 field of view. In most production lines, conveyor motion can be received by the vision detector 130 and used in the vision detector method and apparatus and various purposes disclosed herein, the shaft encoder 132 that generates the signal 140. Tracked by For example, the signal 140 can be used by the vision detector 130 as a reference to the point in time at which the object 112 crosses the fictitious reference point 150, referred to as a recording point.

  The vision detector 130 detects certain events that may occur in the field of view based on appropriate visual conditions. In the exemplary embodiment of FIG. 1, the event corresponds to an accidental labeling by arm 162 and can be detected in a variety of ways as further described below. Images are recorded before, during, and after the event. A human user, such as a manufacturing engineer, can retrieve a recorded image via signal 142 to retrieve a recorded image so that a machine problem with an accidental labeling can be diagnosed. Interact with each other.

  In another embodiment, the material flows continuously through the vision detector rather than individual objects. For example, on the web, there are examples provided below. For the purposes of event detection, recording and retrieval, there is little difference in the continuous flow of individual objects and materials.

  FIG. 2 shows a timeline illustrating a typical operating cycle of a vision detector configured to detect events and record images. A box labeled “c”, such as box 220, represents an image acquisition. A box labeled “a”, such as box 230, represents an analysis (the analysis step is further divided and described below). For example, the next image acquisition “c” preferably overlaps the current image analysis “a” so that the analysis step 230 analyzes the image acquired in the acquisition step 220. In this timeline, the time taken for the analysis is shown to be shorter than the acquisition, but in general, depending on the details of the application, the time taken for the analysis may be shorter or longer than the acquisition.

  If acquisition and analysis overlap, the speed at which the vision detector analyzes and analyzes the image is determined by the time it takes longer between acquisition time and analysis time. This is the “frame rate”.

  The portion 220 of the timeline corresponds to the first event and includes the acquisition and analysis of five event frames. The second portion 202 corresponds to the second event and includes three event frames.

In the exemplary embodiment described herein, analysis of acquired images for event detection includes three main sub-steps as follows.
A visual analysis step to evaluate evidence that an event has occurred or has occurred in one frame. A sequence called a “candidate frame” where the detected event may have occurred. An activity analysis step for identifying the set of frames that have been detected, an event analysis step for determining whether the detected event has occurred in the set of candidate frames.

  Each visual analysis step examines evidence indicating that an event has occurred in a separate frame. Frames with obvious evidence are called “active”. The analysis step of the active frame is indicated by a thick frame line as in the analysis step 240, for example. In the exemplary embodiments discussed herein, this evidence is calculated by image analysis operations of one or more regions of interest of the frame, as disclosed below and in the visual detector methods and apparatus. It is represented by a fuzzy logic value called event detection weight (discussed further below).

  Each activity analysis step considers evidence that an event has occurred or has occurred within the most recent frameset. In the exemplary embodiments discussed herein, event detection is clearly “active” or there is little such evidence that there is some evidence that an event is occurring It is considered to be in the “inactive state” indicating One function of the activity step is to determine this state. In the exemplary embodiment, the transition from the inactive state to the active state occurs when an active frame is discovered. The transition from the active state to the inactive state occurs when a certain number of consecutive inactive frames are found. Candidate frames include a continuous set of frames starting with the first active frame and ending with the last active frame, and may include inactive frames in between. Another function of the activity analysis step is to collect data describing a set of candidate frames.

  Each event analysis step then examines evidence indicating that an event occurred during the candidate frame by reviewing the evidence collected by the previous activity analysis step. In the exemplary embodiment discussed herein, an event analysis step is performed each time a transition from an active state to an inactive state occurs. A candidate frame is considered an event frame if data describing the candidate frame is confirmed and sufficient evidence is found to conclude that an event has occurred. Recording and other appropriate actions are performed as described below.

  Within the scope of the present invention, various methods are used to perform visual, activity and event analysis. Some are described below and in the visual detector methods and apparatus, but many others will be apparent to those skilled in the art.

  In the example of FIG. 2, event detection for the first event 200 enters an active state with a first active frame corresponding to analysis step 240 and with two consecutive inactive frames corresponding to analysis steps 246 and 248. End. The single inactive frame that is the first event corresponding to the analysis step 242 is not sufficient to enter the inactive state.

  When entering the inactive state, for example, at the end of the analysis step 248, an event analysis step is performed. The candidate frames are five event frames that begin at analysis step 240 and end at analysis step 241. In this example, the event analysis step produces a result that an event has occurred, resulting in the recording of the first set of recording frames 210. As a result of a similar analysis of the second event 202, a second set of recording frames 212 is recorded.

  As discussed below, by examining the position of the object in the active frame as it passes through the field of view, the vision detector estimates recording times 250 and 252 that represent the exact time that the event occurred. The time stamp is stored for each recorded frame and indicates the relative time between the recording time and the shutter time corresponding to each such recorded frame.

  When the transition to the inactive state occurs, the vision detector may enter an idle step, such as the first idle step 260 and the second idle step 262, for example. Such a step is optional but may be preferred for several reasons. If the minimum time between events is known, there is no need to look for events until just before a new event can occur. The idle step eliminates false event detection opportunities when an event cannot occur and extends the life of the lighting system because the light can be turned off during the idle step.

  FIG. 3 shows a flowchart that provides details of the event detection analysis steps. A box with a broken line frame represents data used by the flowchart. Squares with rounded corners surrounding the flow chart blocks indicate analysis partitions such as visual analysis step 310, activity analysis step 312 and event analysis 314.

  The active flag 300 stores the active / inactive state used by the activity analysis step. Data collected by the activity analysis step is stored in active data 302 and inactive data 304. The value of the active frame is added directly to the active data 302. In the case of inactive frames, the active analysis step is still unknown because whether the frame is part of the set of candidate frames depends on the state of the later frame. For this reason, the value of the inactive frame is added to the inactive data 304. If an active frame is subsequently found before the transition to the inactive state occurs, the inactive data 304 is added to the active data 302 and cleared. Inactive data 304 remaining at the time of transition to the inactive state is discarded. Both data contain the number of frames added.

  In the exemplary embodiment of FIG. 3, the flowchart is executed many times, from acquisition block 320 to continuous block 324, once in each frame. The acquisition block 320 synchronizes the analysis by frame acquisition and duplicates the acquisition of the next frame with the analysis of the current frame. The visual analysis block 322 performs a visual analysis step and calculates an event detection weight d from the analysis of the acquired image.

Next, an activity analysis step 312 is performed. Active block 330 tests active flag 300 to determine the current state of event detection. If event detection is inactive, the first threshold block 340 determines the active / inactive status of the current frame by comparing _d against the threshold td. Be greater than the threshold t _d event detection weight d, the frame is inactive, active analysis ends at the current frame. If event detection weight d is greater than threshold t _d , the frame is active, active transition block 342 sets active flag 300, and initialization block 344 initializes active data 302 using the value of the current frame. Inactive data 304 is cleared.

If determined that the active block 330 is active the current state, the second threshold block 346, by comparing the d threshold t _d, to determine the active / inactive state of the current frame. If d is not greater than the threshold, the frame is inactive and the count test block 350 refers to the frame count of the inactive data 304 to determine if more than the parameter k of consecutive inactive frames has been found. If not found, the inactive update block 354 updates the inactive data 304 by adding a value from the current frame (including increasing the frame count by one). If so, the inactive transition block 352 clears the active flag 300 and execution continues to the event analysis step 314.

  If second threshold block 346 determines that the current frame is active, active update block 360 updates active data 302 by adding values from both the current frame and inactive data 304. . Then, the clear block 362 clears the inactive data 304.

  If the activity analysis step 312 transitions from an active state to an inactive state, an event analysis step 314 is performed. Condition block 370 tests active data 302 to determine if an event has occurred. If not, the collected data is ignored and execution continues. If so, event block 372 records a frame for recording, as further described below. The idle block 374 waits for the idle period before continuing.

  FIG. 4 further illustrates the analysis steps of the exemplary embodiment and is used in conjunction with the timeline of FIG. 2 and the flowchart of FIG. 3 to understand the basic operation of the present invention.

FIG. 4 illustrates how evidence is weighted for event detection in an exemplary embodiment. As described above, information that constitutes evidence that an event has occurred or has occurred in the visual field is called “event detection weight”. The figure shows the coordinates of the event detection weight d _{i on} the vertical axis 400 and the frame count i on the horizontal axis 402. Each frame is represented by a vertical line such as a straight line 426. The frame count is an arbitrary integer.

In this example, d _i is a fuzzy logic value representing evidence that an event is occurring in frame i and uses the method described further below and in the method and apparatus of the visual detector. And calculated by the vision detector of each frame.

In the exemplary embodiment of FIG. 4, the event detection threshold t _d is 0.5, so frames with d _i greater than or equal to 0.5 are considered active frames. For reference, a line 430 with d _i = 0.5 is drawn. The event detection weight for the active frame is drawn as a black circle, as at point 410, and the event detection weight for the inactive frame is drawn as a white circle, as at point 416.

  In the example of FIG. 4, event detection enters the active state at frame 422 and four consecutive inactive frames are seen (in this example, the inactive frame count threshold k = 3) after frame 424, the inactive state. to go into. The set of candidate frames begins at frame 422 and ends at frame 426. The isolated inactive frame 420 does not cause a transition to the inactive state.

  FIG. 5 shows details of the data collected by activity analysis step 312 and used by event analysis step 314 in the exemplary embodiment, as well as the example of FIG. In this embodiment, active data 302 and inactive data 304 contain sufficient information to calculate these data when a transition to the inactive state occurs. Each of the data in FIG. 5 includes a symbol 500, a description 510 of the function that defines the value, described further below, and an example 520 that indicates what the example values of FIG. 4 are.

The above description of a method for weighting evidence to determine whether an event has been detected is intended as an example of a useful embodiment, but is not limited to a method that can be used within the scope of the present invention. For example, the example constant t _d = 0.5 used above may be replaced with any suitable value. Many methods of visual event detection are added by those skilled in the art.

[Example of equipment]
FIG. 6 shows a high level block diagram for a visual detector used for visual detection, recording and retrieval of events. The visual detector 600 can be connected to a suitable automated instrument 610 and it can include a PLC, a reject actuator, a rotary encoder, and / or a photodetector using the signal 620. . These connections do not require the detection, recording, and retrieval of events, but can be useful, for example, in cases where it is preferable to use a visual detector for additional purposes, as taught in visual detection methods and devices. it can. For example, there is a particularly preferred way to provide an output pulse to indicate that an event has been detected. For example, the pulses are delayed and synchronized in recording time, as taught in the visual detector methods and devices, and used by PLCs, actuators, or other devices.

  For retrieval of an image display of detected events, the vision detector uses signal 640 to connect to a human machine interface (HMI) 630, such as a PC or portable device. The HMI 630 is also used as a setting. The HMI 630 needs to be disconnected for event detection and recording, but of course it must be connected for searching and display. Signal 640 can be implemented in a format and / or protocol that meets conditions and converted in a wired or wireless manner.

  Event images recorded by the vision detector 600 are also retrieved by the automated image analysis system 650 and include, but are not limited to, conventional machine vision systems. Such a system is possible with visual detectors designed to operate at ultra-high frame rates without the need for human analysis inherent in the use of HMI 630, and is also used for modern analysis of images. obtain.

  FIG. 7 shows a block diagram of an illustrative example of a visual detector that can be used to practice the present invention. The digital signal processor (DSP) 700 launches software to control acquisition, analysis, recording, HMI communication, and some appropriate functions required by the visual detector. The DSP 700 is connected to a memory 710 that includes a program, data for holding programs and configuration information, and a high-speed random access memory for non-volatile memory when the power is removed. The memory 710 holds a recorded image for subsequent retrieval. The DSP is also connected to an I / O module 720 that supplies signals to the automatic equipment, the HMI interface 730, the illumination module 740, and the imaging device 760. The lens 750 focuses the image on the light sensitive elements of the imaging device 760.

  The DSP 700 is a device capable of digital computation, information storage, and connection to other digital elements, including but not limited to a general purpose computer, PLC, or microprocessor. DSP 700 is inexpensive but is preferred because it is fast enough to operate at high frame rates. It is more preferable because it has image analysis and can simultaneously receive and store pixel data from the imaging device.

  In the illustrative example of FIG. 7, DSP 700 is ADSP-BF531, produced at Analog Devices, Norwood, Massachusetts. The parallel peripheral interface (PPI) 770 of the ADSP-BF531 DSP 700 receives pixel data from the imaging device 760 and sends the data to the memory controller 774 through a direct memory access (DMA) channel 772 for storage of the memory 710. Under appropriate software control, the use of PPI 770 and DMA 772 allows images to be acquired simultaneously with other analyzes performed by DSP 700. Software instructions for controlling with PPI 770 and DMA 772 are included in the ADSP-BF533 Blackfin Processor Hardware Reference (Part No. 82-002005-01) and Blackfin Processor Instruction Reference (Part No. 82-000410-14) Implemented by those of ordinary skill in the art and incorporated by reference to both herein. It should be noted here that the ADSP-BF531, competing ADSP-BF532 and ADSP-BF533 devices have the same program instructions and are illustrative of this in order to obtain the appropriate price / execution transaction. It is used alternately in the example.

  The preferred high frame rate by the visual detector encourages the use of an imaging device that is not the imaging device used in prior art vision systems. The imaging device is preferably unusually sensitive to light so that it can be operated with inexpensive illumination and with very short shutter times. More preferably, the pixel data can be digitized and transferred in a DSP that is even faster than prior art vision systems. It is still preferred that it is not expensive and has a global shutter.

  These objects can be accommodated by choosing an imaging device that is more sensitive and has a lower resolution than the imaging devices used in prior art vision systems. In the illustrative example of FIG. 7, imaging device 760 is an LM9630 manufactured by National Semiconductor of Santa Clara, California. The LM9630 has an array of 128 × 100, total 12800 pixels, approximately 24 times less than typical prior art systems. Each pixel is 20 square microns wide with high photosensitivity. The LM9630 can deliver 500 frames per second if the set with a shutter time of 300 microseconds is sensitive enough (in most cases) to use LED illumination and bring it to 300 microseconds. This resolution is considered much too low for the vision system, but is sufficient for the detection features that are objects of the method and apparatus of the vision detector. The electrical interface and software control of the LM9630 is performed by those skilled in the art according to the instructions included in the January 2004 Rev 1.0 LM9630 data sheet, which is incorporated by reference to both here. It is.

  The illumination 740 is not expensive, but preferably has sufficient brightness for a short shutter time. In an illustrative embodiment, a string of bright red LEDs operating at 630 nanometers uses, for example, HLMP-ED25 manufactured by Agilent Technologies. In another embodiment, bright white LEDs are used to perform the preferred illumination.

  In the illustrative example of FIG. 7, I / O module 720 provides output signals 722 and 724 and input signal 726. The input signal 726 is used for event detection with a logical connection appropriate as taught in the visual detector method and apparatus.

  The next term image acquisition device means acquiring and storing a digital image. In the illustrative example of FIG. 7, the image acquisition device 780 comprises a DSP 700, an imaging device 760, a memory 710, and associated electrical interfaces and software instructions.

  The next term analyzer provides a means of analysis of digital data, including but not limited to digital images. In the illustrative embodiment of FIG. 7, the analyzer comprises a DSP 700, a memory 710, and associated electrical interfaces and software instructions.

  The next term output signal provides a means for producing an output signal corresponding to the analyzer. In the illustrative example of FIG. 7, the output signal device 784 includes an I / O module 720 and an output signal device 722.

  The following terminology is directed to several purposes and is referenced to an organized set of operations performed by the preferred device to perform the operations targeted by the mechanism, device, component, software, or firmware. This includes, but is not limited to, any composition that works together in one place or at various places.

In the illustrative embodiment, the various processes used in the present invention are performed by collecting interactions between digital hardware elements and computer software instructions. Examples of hardware elements include the following.
A DSP 700 that performs general information processing operations under the control of preferred computer software instructions
Memory 710 that provides storage and retrieval operations for images, data, and computer software instructions
An imaging device 760 that is supplied in combination with other elements, such as the image acquisition operation described herein.
I / O module 720 that provides interface and signal operations
An HMI interface 730 that functions as a human machine interface.

In one embodiment, computer software instructions for performing the operations described herein include, for example, the following software instructions.
The flowchart steps of FIG. 3 illustrating the illustrative acquisition and analysis process portion. The fuzzy logic elements of FIG. 8 illustrating the example decision logic. The diagram illustrating the set of software elements used to implement the invention. Graphical controls of FIGS. 10 and 20 showing nine software elements and how a human user can select parameters for operation.

  Furthermore, it will be appreciated by those skilled in the art that the above is only a list of examples. It is not complete, and preferred computer software instructions can be used in an illustrative embodiment to perform the processing used for any of the diagrams described herein.

Examples of processing described in this specification include the following.
An acquisition process performed by the image acquisition device 780, including the acquisition block 320 (FIG. 3) and other operations as described herein. The analyzer 782 and preferred software elements shown in FIG. 3 and various analysis processes, including the portions of FIG. 3, and other operations described herein, such as visual analysis step 310, activity analysis step 312, and event analysis step 314, shown in FIG. 9. Various selection processes, including event block 372 and other operations described herein, performed by analyzer 782 and the appropriate software elements, analyzer 782 shown in FIG. Various decision processes, including condition block 370 and other operations described herein, performed by software elements.

  As will be apparent to those skilled in the art, there are numerous alternative configurations, devices, and software instructions that can be used within the scope of the present invention to implement image acquisition device 780, analyzer 782, and output signaler 784. . Similarly, there are a number of software instructions that can be used within the scope of the present invention to perform the processes described herein.

[Fuzzy logic decision making]
FIG. 8 illustrates the fuzzy logic elements used in the illustrative example to review and determine evidence, including determining whether an event is occurring or has occurred.

  A fuzzy logic value is a number between 0 and 1 that indicates an estimate of the confidence that a particular state is true. A value of 1 indicates a high confidence that the state is true, 0 indicates a high confidence that the state is false, and an intermediate value indicates an intermediate level of confidence.

  The more familiar binary logic is a subset of fuzzy logic where the confidence value is limited from 0 to 1. Thus, the embodiments described herein that use fuzzy logic can use those values that are replaced by an equivalent binary logic method or device and can be used as another binary logic value in a fuzzy logic method or device. .

  A fuzzy logic value is obtained by using a fuzzy threshold, as a binary logic value is obtained from an original measurement value by using a threshold value. Referring to FIG. 8, a graph 800 shows a fuzzy threshold. The X-axis 810 shows the original measurement, the f-axis 814 shows the fuzzy logic value, the range of x is a function where all possible original measurements and the range of f is 0 ≦ f ≦ 1. .

In the illustrative embodiment, the fuzzy threshold comprises two numbers, indicated by the x-axis, low threshold t ₀ 820, and high threshold t ₁ 822, corresponding to points on functions 824 and 826. The fuzzy threshold can be defined by the following equation.

It should be noted here that this function works well when t ₁ <t ₀ . Other functions can also be used as fuzzy thresholds, such as sigmoids.

  Here, t and σ are threshold parameters. In a specific example where the goal is simplicity, a conventional binary threshold can use a binary logical value as a result.

  Fuzzy decision making is based on fuzzy versions of AND840, OR850, NOT860. The fuzzy AND of two or more fuzzy logic values is the minimum value and the fuzzy OR is the maximum value. The fuzzy NOT of f is 1-f. If the fuzzy logic value is limited to 0 or 1, the fuzzy logic value is equal to the binary logic value.

  In an illustrative example, whenever a true / false decision is required, a fuzzy logic value is considered true if it is at least 0.5, and false if it is 0.5 or less. It is believed that there is.

  As will be apparent to those skilled in the art, there is no criticality for the values 0 and 1 as used in connection with the fuzzy logic described herein. Use as an intermediate value to indicate an intermediate level of trust, use any number to indicate that the state is true to indicate confidence, and use a different number to indicate confidence that the state is false. Can do.

[Software elements of the present invention]
FIG. 9 shows the organization of a series of software elements (eg, program instructions on a computer readable medium) used by an example for frame analysis, determination, input sensing, and output signal conditioning. Elements may be implemented using a class hierarchy of a conventional object-oriented programming language such as C ++ so that each element corresponds to a class. However, any programming technique and / or language that meets the criteria can be used to perform the processes described herein.

  As shown, a class with a dotted border, such as the gadget class 900, does not exist independently, but is an abstract base class used to construct concrete derived classes, such as the locator class 920. . A solid bordered class represents a dynamic object that can be created and destroyed using the HMI 630 when the user needs to set up an application. Classes bounded by dashed lines, such as input 950, represent static objects associated with specific hardware and software resources. Static objects are always present and cannot be created or destroyed by the user.

  Since all classes are derived from the gadget class 900, all objects that are instances of the class shown in FIG. 9 are a kind of gadget. In a specific example, each gadget is: 1. has a name that the user can select; 2. Other gadgets have a logic output (fuzzy logic) that can be used as a logic input to control decision and output signals; 3. has a set of parameters that can be configured by the user to specify the operation; 4. has one such parameter that can be used to invert the logic output (ie fuzzy NOT); It can be executed and its parameters, logic input if applicable, and for certain gadgets the logic output is updated based on the contents of the current frame, and can cause side effects such as setting the output signal.

  The frame analysis task includes executing each gadget once in a fixed order to ensure that all logical inputs to the gadget have been updated prior to gadget execution. In some embodiments, the gadget is not executed during a frame whose logic output is not required.

  The photo class 910 is the base class for all gadgets whose logic output depends on the content of the current frame. These classes are classes that actually perform image analysis. Each picture measures certain features of the region of interest (ROI) of the current frame. The ROI corresponds to the visual function of the object being inspected. This measurement is called the analog output of the photograph. The logical output of the photograph is calculated from the analog output using a fuzzy threshold that exists in a series of parameters that can be configured by the user, called the sensitivity threshold. The logical output of the photo can be used to provide evidence that is used in making decisions.

  Detector class 930 is a basic class of photographs whose primary purpose is to provide evidence in making ROI measurements and decisions. In a specific example, all detectors ROI are circular. A circular ROI simplifies execution because there is no need to handle rotation, and only one ROI shape simplifies what the user has to learn. Detector parameters include position and ROI diameter.

  The luminance detector 940 measures the weighted average and percentile luminance in the ROI. A contrast detector 942 measures the contrast at the ROI. The edge detector 944 measures the region where the ROI looks like an edge in a particular direction. Spot detector 946 measures the area where the ROI appears to be a round feature such as a hole. Template detector 948 measures the region where the ROI looks like a pre-trained pattern selected by the user. The operation of the detector is further described in the visual detector method and apparatus.

  Locator class 920 represents a photograph with two main purposes. The first is to generate a logic output that can provide evidence in making a decision, which can be used like any other detector. Secondly, by determining the position of the object in the field of view of the visual detector, the position of the ROI of the other photo can be moved to track the position of the object. Any locator can be used for one or both purposes.

  In a specific example, the locator searches for an edge in a one-dimensional region within the frame. The search direction is perpendicular to the edge and is among the parameters set by the user. The analog output of the locator is similar to the analog output of the edge detector. The locator is further described in the visual detector method and apparatus.

  In other embodiments, the locator searches the multi-dimensional search range using known methods including transformation, rotation, and size of freedom. Suitable methods include those based on normalized correlations, generalized Hough transforms, geometric pattern repairs, all known techniques that have been commercially available for many years. Specific examples of multidimensional locators can be found in Brian Mirtic and William M., filed on Nov. 2, 2004. A co-pending US Patent Application No. 10 / entitled “Method for Setting Visual Detector Parameters Using Production Line Information”, a teaching method by Silver that is specifically incorporated herein by reference. 979,535.

  Input class 950 represents an input signal sent to a visual detector that is used to influence event detection. The output class 952 represents an output signal from a visual detector that may be used to notify a PLC or actuator that an event is detected. In an illustrative implementation, for each physical input, there is one static instance of the input class, such as a typical input signal 726 (FIG. 7), and a typical for each physical output. There is one static instance of the output class, such as output signals 722 and 724. As taught in visual detection methods and devices, external automated devices can determine when to use the delay time or where to use the encoder count when an event occurs. The output can form a delayed pulse that synchronizes the recording time.

  The gate base class 960 performs fuzzy logic decision making. Each gate has one or more logic inputs that can be connected to logic outputs of other gadgets. Each logic input can be inverted (fuzzy NOT) using a user-configurable parameter. The AND gate 962 performs a fuzzy AND operation, and the OR gate 964 performs a fuzzy OR operation.

  Decision class 970 is the base class for objects that consider evidence on successive frames to make decisions. In an illustrative example of the present invention, activity analysis step 312 and event analysis step 314 (visual analysis step 310 is performed by composition of photographs, inputs, and / or gates used in the examples given below. Event detection judgment 972 having the purpose of implementing Other types of judgments are taught in visual detection methods and devices by being present in preferred embodiments for combining the functions given therein and event detection as given herein.

  Each decision has a logic input to connect the logic output of a typical gate, giving the user a logic combination of gadgets, which are photos, or mostly photos and other gates. The event detection decision logic input provides an event detection weight to each frame. The specific example of the present invention is an example given below, and it is clearly conceivable that it may be used rather than event detection judgment.

  The logic output of the event detection decision provides a pulse that indicates when an event has been detected. A pulse rise occurs when event analysis step 314 detects an event, such as at the end of analysis step 248 of FIG. 2, and a fall occurs after, for example, the end of idle step 260.

  FIG. 10 shows a graphical control displayed on the HMI for the user to see and manipulate to set parameters for event search decisions. The set of graphical controls displayed on the HMI 630 for setting gadget parameters is referred to as a parameter view.

  The name text box 1000 allows the user to browse and input a name for this event detection determination. The time display 1002 indicates the time taken for the latest activation of this event detection determination. The logical output label 1004 indicates the current logical output value of this event detection determination, and is a color, shape, or other indicator for distinguishing whether it is true (≧ 0.5) or false (<0.5). May change. An inversion check box 1006 causes a logic output of this event detection judgment to be inverted. It should be noted that the name text box 1000, time display 1002, logic output label 1004, and inversion check box 1006 are displayed in the parameter view for all gadget types, as further described in the visual detection method and apparatus. Is to be general.

  The idle time spinner 1020 allows the user to specify a time interval, as indicated by the idle step 260 (FIG. 2) and the idle block 374 of FIG. The erasure frame spinner 1030 allows the user to specify the maximum number of consecutive invalid frames that will be accepted without the activity analysis step 312 converting to an invalid state. The value specified by erasure frame spinner 1030 is used as parameter k in count test block 350 of FIG.

  Marking control 1040 allows the user to select between flow and stroke events to calculate the recording time, as further described individually. In order to calculate the recording time, the user must identify the locator using the locator list control 1042.

  When an event is detected, the recording interval control 1050 allows the user to specify the time interval at which the image is recorded in relation to the recording time. In an alternative embodiment, not shown, the user specifies whether an event frame has been recorded and the number of frames before and after the event frame is recorded.

  The state text 1010 is tested by the state block 370 and allows the user to identify the event state that was used by the event analysis step 314 to determine if an event occurred. In the example of FIG. 10, the state text 1010 is used by a conventional programming language such as C, and includes a text string indicating a logical expression with a similar syntax. In order to specify the calculation of the truth value from the activity statistics 302, the representation is expressed in the elements of the activity statistics 302, using numerical constants, logic, comparisons, arithmetic operators, punctuation such as parentheses, 5 to the symbol 500 are combined. A method of calculating a true / false value based on such a text character string is a known technique.

  In the illustrative example of FIG. 10, an event occurs if there are at least two candidate frames and the average event detection weight is 0.50 or less, or 0.75 or more. An example where this is useful is shown in FIGS. 15 and 16 described below.

[Useful examples]
FIG. 11 shows an example of how a photo is used to detect an event corresponding to an object using a non-applied label, such as the non-applied label 122 (FIG. 1). FIG. 11 shows an image of the object 1100 corresponding to the object 116 from FIG. 1 and has a superimposing graphic representing a photograph together with a label mechanism 1110, which is displayed on the HMI 630 so that the user can see and manipulate it. . Display of images and superimposed graphics on the HMI is called image display.

  The locator 1120 is used to detect the upper end of the object and confirm the position, and another locator 1122 is used to detect the right end and confirm the position. A luminance detector 1130 is used to help detect the presence of an object. In this example, the background is brighter than the object and the sensitivity threshold is set to distinguish between the two brightness levels, and the logic output is inverted to detect darker objects rather than brighter backgrounds. .

  As described further below, locators 1120 and 1122 and luminance detector 1130 together provide the necessary evidence to determine the presence of an object. Clearly, the event corresponding to “object with non-applied label” does not occur if no event exists.

  Edge detector 1160 is used to detect the presence and position of label 1110. An edge detector if the label is not present, is misplaced horizontally, rotates significantly, has a bent cammer as shown, or is otherwise not applicable The analog output is very low. Of course, the label 1110 is often unapplied, not detected by the edge detector 1160, so other photos may be used when a failure needs to be detected in any given production situation. is there.

  For example, the luminance detector 1150 is used to check whether the correct label has been applied. In this example, the correct label is white and the wrong label is dark.

  As the object moves from left to right through the visual detector field of view, the locator 1122 tracks the right edge of the object and is in the correct position with respect to the object. Reposition instrument 1160. The locator 1120 repositions the detector based on the position of the top edge of the object and corrects for any vertical position variations of the object in the field of view. In general, the locator can be placed at any position.

  The user can manipulate the photos within the image display using known HMI techniques. A photo can be selected by clicking the mouse, and by dragging, the ROI can be moved, resized, and rotated. Additional operations on the locator are described in the visual detector method and apparatus.

  FIG. 12 shows a logical view including a wiring diagram corresponding to the setting example of FIG. The wiring diagram shows all the gadgets used to detect events and interfaces to automated equipment and the connections between the gadget's logic inputs and outputs. The wiring diagram is displayed on the HMI 630 for the user to see and operate. The display of gadgets and their logical interconnections in the HMI are called logical views.

  Still referring to the wiring diagram of FIG. 12, locator 1220 named “top” is connected to AND gate 1210 by wire 1224 corresponding to locator 1120 in the image display of FIG. Similarly, “side” locator 1222 corresponds to locator 1122 and “box” detector 1230 corresponds to luminance detector 1130 and is wired to AND gate 1210. As indicated by the small circle 1232 and as described above, the “box” detector 1230 is inverted to detect darker objects against a light background.

  In the wiring diagram, the luminance detector “label” 1250 corresponds to the luminance detector 1150 and the edge detector “label edge” 1260 corresponds to the edge detector 1160 and is wired to the AND gate 1212. The logic output of the AND gate 1212 is converted and wired to the AND gate 1210 to represent the level of reliability that the label 1210 is not applicable.

  The logic output of AND gate 1210 represents the presence of an object with a level of confidence, indicating that level, ie, the level of confidence that the event occurs, is not applicable. The logic output of AND gate 1210 is wired to event detection decision 1200 because it is used as an event detection weight for each frame. Many alternative event states are also more appropriate depending on the situation of the application, but the event states of the appropriate event detection decision 1200 in this configuration are “n> = 3 and m> = 0.5”.

  Selection of a gadget to be wired for event detection determination is performed by a user based on application knowledge. In the example of FIGS. 11 and 12, the user may have determined that detecting only top and right is not enough to confirm the presence of an object. Note that the locator 1122 may react to the left edge of the label in the same way that it reacts to the right edge of the object, and perhaps at this point in the production cycle, the locator 1120 is in the background. This means that you may have found another edge. By adding the detector 1130 and satisfying all three conditions using the AND gate 1210, the detection of the event is reliable.

  If an event is detected, the image may be further recorded as described below. Obviously, an image corresponding to time before the event frame is most likely to show exactly how the label is unapplied. It is clearly preferred in this example that the visual detector be placed as close as practical to where the label is applied so that the object can be seen.

  The logic output of the event detection decision 1200 is wired to an output gadget 1280 named “Signal” that, when connected to an automatic device, such as a PLC or actuator, optionally from a visual detector. Control the output signal. The output gadget 1280 is further appropriately configured by the user as described in the visual detector methods and apparatus. Since the automatic device can determine when, when an event occurs, when to use time and where to use the encoder count, the output gadget 1280, as taught in the visual detector method and apparatus, A delay pulse for synchronizing the recording time can be formed.

  The user can manipulate the gadget within the logical view by using known HMI techniques. Gadgets can be selected by clicking the mouse, their positions can be moved by dragging, and wires can be created by dragging and dropping.

  Those skilled in the art will recognize that various events can be detected by appropriate selection, configuration, and wiring of gadgets. Those skilled in the art will also recognize that the gadget class hierarchy is the only of many software technologies used to implement the present invention.

  FIG. 13 shows an image display corresponding to another configuration of a visual detector for detecting events that may be useful in the production setup shown in FIG. In this example, the event occurs when the display arm 162 is fully extended (the tip of its stroke), but the object 1300 is in an incorrect placement to receive the label 1310. The arm edge detector 1340 is disposed at a position in the field of view corresponding to the tip of the stroke of the display arm 162. Note that this position is fixed relative to the field of view, and it does not move on the production line, so there is no need to use a locator. Top edge detector 1320 and side edge detector 1330 are used to confirm that object 1300 is in a preferred position at the tip of the stroke.

  FIG. 14 is a logic diagram showing the configuration of the gadget corresponding to the image display of FIG. 13 in order to detect an event in which an object corresponds to the wrong position at the tip of the stroke of the display arm 162. The “arm” 1440 corresponds to the arm edge detector 1340, the “top” 1420 corresponds to the top edge detector 1320, and the “side” 1430 corresponds to the side edge detector 1330.

  Using the AND gate 1410 wired with the converted AND gate 1412, as shown in FIG. 14, the event detection decision 1400 determines that the object 1300 has a top edge detector 1320 when the arm 162 is at the tip of a stroke. And an event detection weight representing a level of confidence that the position is not specified by at least one of the side edge detectors 1330. In the case of this configuration, the event state of the appropriate event detection determination 1400 is “n> = 2”.

  If an event is detected, the image may be further recorded as described below. In addition, the logic output of event detection decision 1400 is wired to an output gadget 1480 named “Signal”, which, when connected to an automated device, such as a PLC or actuator, detects visual detection as needed. Control the output signal from the instrument. The output gadget 1280 is further appropriately configured by the user as described in the visual detector methods and apparatus.

  FIG. 15 shows a logical display corresponding to an image display that forms yet another configuration of a visual detector for detecting events that may be useful in the production setup shown in FIG. In this example, the display arm 162 event is not expanded or the display arm 162 is overextended if the stroke tip is in the wrong place, or it has been extended to the correct position but the correct label application. If it is too short or too long to be there, an event occurs.

  Excessive extension is easy to detect. The edge detector 1512 is located below the expected tip of the display arm 162 downward stroke. The corresponding logical display edge detector “Hyper” 1540 is wired to the “Hyper Event” event detection decision 1570, which may use the event state “w> = 0.95” to detect an arm that is overextended. There is. Note that using the aggregate event detection weight w, this event state accepts a single frame as sufficient evidence if the event detection weight for that frame is very reliable, and the event detection weight is low When showing reliability, it requires at least two frames.

  In other states, the locator 1500 is arranged to detect that the display arm 162 is within the range of the position near the tip, and the edge detector 1510 is used to detect that the display arm 162 is the tip. Be placed. Corresponding logical display locator “Stroke” 1520 and inverted edge detector “Apex” 1530 are wired as shown in AND gate 1550, which in turn is routed to event detection decision “Stroke Event” 1560. Is done. In this configuration, the event detection weight represents the level of reliability that the display arm 162 is near but not the tip of the stroke.

  Note that the logical representation of FIG. 15 includes the use of two event detection decisions. If any of the decisions finds sufficient evidence, the event is detected, so if more than one event detection decision is used, each is operated independently. Each decision uses a copy of the activity flag 300, valid statistics 302, invalid statistics 304 and manipulates its own valid analysis step 312 and event analysis step 314. It should be noted that the visual analysis step 310 is operated by other gadgets such as photographs and gates.

FIG. 16 shows how events are clearly shown, for example, for the event detection decision “stroke event” 1560, which is the appropriate structure of FIG. Four plots of event detection weights d _i vs. frame count I, similar to the plot shown in FIG. 4 described above, are shown.

  The first plot 1600 shows an unexpanded arm. The arm moved closer to the required tip in about 12 frames but did not actually reach the tip. The second plot 1610 shows an arm that has extended to the required tip for the correct label application, but only one frame has remained too short there. The third plot 1620 shows the arm extended correctly, reaching the required tip and remaining there for about three frames. Fourth plot 1630 shows the arm extended longer, reaching the required tip and remaining there for about seven frames.

  The event state “n> = 2 and (a <.50 | a> .75)” in FIG. 10 is suitable for detection of the first plot 1600, the second plot 1610, and the fourth plot 1630, but the correct arm Is not detected in the third plot 1620 corresponding to the extension of. The term “a <.50” is detected in the first plot 1600 and the second plot 1610. The term “a> .75” is detected in the fourth plot 1630. The term “n> = 2” promises that no single frame false event will be detected.

  Obviously, there are many other configurations and event conditions that are preferred for detecting erroneous extension of mechanical equipment, such as display arm 162, and would occur to those skilled in the art. FIG. 17 illustrates one method of constructing the invention for detecting and recording an image of a defect on a continuous web. Image display 1710 shows a portion of continuation web 1700 passing through a visual detector.

  Locator 1720 and edge detector 1722 are configured to inspect the web. If the web is broken, folded, or substantially frayed at either end, the locator 1720 and / or edge detector 1722 will produce a false output (logic <0.5). Let's go. If the web moves up and down, the locator 1720 will include an edge detector 1722 in the relevant right position to track the top edge or detect the bottom edge. However, if the width of the web changes substantially, the edge detector 1722 will produce a false output.

  In the logical representation, “top” locator 1740 represents locator 1720 and “bottom” locator 1750 represents edge detector 1722. These are connected to an AND gate 1760 and its logic output is inverted and wired in event detection decision 1770.

[Marking, stroke event, synchronized output]
FIG. 18 shows a timing diagram used to explain how the output signal of the vision detector may be synchronized with the recording time. Signal synchronization is desirable for various industrial inspection purposes, such as control of downstream actuators.

  Since visual event detection is a new function, a new output signal control is proposed. The vision detector should be able to control some external actuators either directly or by functioning as an input to the PLC. This is because the timing of the output signal is some physical means, such as when the object passes a certain fixed point in the production flow (flow event) or when the machine component reaches the top of the stroke (stroke event). Suggests that it can be related to a certain time with a certain degree of accuracy. In the example of FIG. 1, the fixed point can be the recording point 150, and in the timeline of FIG. 2, the times are the recording times 250 and 252. In the example of FIG. 15, the edge detector 1510 is located at the apex of the stroke of the label attaching arm 162. In FIG. 18, the time is the recording time 1800. Encoder count may be used instead of time.

  The present invention can provide an output that is synchronized with recording time with a certain degree of accuracy, whether directly controlling the actuator or being used by a PLC or any other external device. One problem, however, is that the present invention detects an event several milliseconds after it occurs, that is, milliseconds after the recording time. Moreover, the delay, if not so, when recording time occurs in the acquisition / analysis cycle can vary greatly depending on how many frames are analyzed.

  FIG. 18 shows the event detection logic output 1840. Detection pulse 1870 is displayed on event detection logic output 1840 when a decision is made at decision point 1810. Decision point 1810 corresponds to the point in time when event block 372 in the flowchart of FIG. 3 is executed. The decision delay 1830 from the recording time 1800 to the decision point 1810, if not so, can vary when the recording time occurs in the acquisition / analysis cycle, depending on the number of frames analyzed. is there. Thus, the timing of the detection pulse 1870 does not convey accurate information about when the event occurred.

  The problem of variable decision delay 1830 applies to all devices that attempt to detect events by acquiring and analyzing images. Further, the present invention is applied to a case where it is preferable to provide a signal indicating when an event has occurred with an accuracy higher than the frame period (the reciprocal of the frame rate). The present invention solves this problem by measuring the recording time 1800 and then synchronizing the output pulse 1880 of the output signal 1860 to the recording time.

  The output pulse 1880 is generated with a fixed output delay 1820 from the recording time 1800. The recording time measurement operation is called “marking”. The recording time, when determined by an appropriate detector, is the time known to acquire an image, a known position of the object, a machine component or an object moving in the field of view (or an encoder count). ) Can be determined with much higher accuracy than linear interpolation, least squares fit, or other well known methods. The accuracy depends on the shutter time, the total time of the acquisition / analysis cycle, the speed of movement, and other factors.

  In the exemplary embodiment, the user selects one detector whose search range is substantially the same as the direction of travel used for marking. In the case of a flow event, the mark point is arbitrarily selected as the center point of the detector range as described above. However, as long as the recording point is fixed, the fictitious position does not matter. Reference point. The user can achieve a favorable synchronization of the output signal by adjusting the delay from this arbitrary time.

  If an event is detected but the motion does not cross a recording point during an active frame, the recording time may be less accurate because the estimation is based on estimation. In the case of a stroke event, the recorded point is the apex of the stroke and is measured as described below. It will be apparent that other definitions of recording points can be used to practice the present invention.

  Note that the output signal can only be synchronized with the recording time if the output delay 1820 is longer than the maximum expected delay 1830. Thus, all actions taken as a result of the output pulse 1880 must be well downstream of the recording point, eg, operation of the actuator, and is assumed for almost all applications. .

  FIG. 19 shows a graph of detector results as a function of time for stroke events. The detector must be configured to search in a direction substantially parallel to the stroke direction, such as, for example, detector 1500 in FIG. Note that multi-dimensional detectors can also be used as long as the searched dimension includes a direction substantially parallel to the stroke direction. The time measured from any reference point is depicted on the horizontal axis 1900. The logic output of the detector is drawn as a series of square points, such as white points 1920 and filled points 1922, connected by a detector position curve 1950. Only at different times when the square points corresponding to the frames are drawn, the detector measures the position. Accordingly, the detector position curve 1950 is understood to be arbitrarily drawn by the reader and does not represent a continuous measurement by the detector. The logical output value corresponding to the vertical position of the square point is drawn on the first vertical axis 1910. White square points, such as white dots 1920, indicate low logic output values (less than 0.5, baseline 1914 in the exemplary embodiment) when the detector has little confidence that it has found the feature of the subject image. Is equivalent). A black square point, such as a filled point 1922, is a high logical output value (greater than 0.5 in the exemplary embodiment) when there is a strong confidence that the detector has found the extreme of the target image, and thus Indicates that the measurement position is valid.

  The measured position of the detector is drawn as a series of position points drawn as black circles, such as position point example 1930. A position value corresponding to the vertical position of the position point is plotted on the second vertical axis 1912 and measured in pixels from the center of the detector. Note that the position points are displayed only in frames where the detector's logic output is greater than or equal to 0.5, i.e., frames where there is a strong confidence that the detector has found a feature of the target image.

  The mechanical component of the field of view advances the detector's search range by about +6 pixels toward the vertex, about 4 frames, and is maintained at that vertex for about 6 frames before searching. This can be seen by checking a position point that has been retracted by another four frames before the range is exceeded. To calculate a specific recording time for this stroke event, the exemplary embodiment uses the position points to calculate the optimal parabola 1940, and the vertex of the parabola 1960 is easily determined. In the illustrative example, the recording time (the peak of the optimal parabola) occurs at 19.2 milliseconds.

  Methods for calculating an optimal parabola from a set of points are well known in the art. It will be apparent to those skilled in the art that other methods for determining the apex of a stroke event can be used within the scope of the present invention. In addition, in order to determine that the methods described herein can be used to track movements other than the flow and stroke events discussed herein, and to determine the recording time for such movements, It is clear that the appropriate curve can be fitted and other techniques can be used.

  FIG. 20 shows a parameter display for user setting of the output device, such as control for setting the output delay 1820 (FIG. 18). The mode control 2000 allows the user to select how to control the output signal. In “Linear” mode, the logic input is passed directly to the output signal with no delay or synchronization. In “delay” mode, on the rising portion of the logic input, the output pulse is delayed by the amount specified by the delay control 2010 by a time delayed from the most recently measured recording time (or encoder count) and Scheduled to occur for the period specified by control 2020. Scheduled pulses may be placed in a FIFO associated with the output device.

[Recording and searching images]
FIG. 21 shows organizational details of some of the memory 710 (FIG. 7) used in the exemplary embodiment. Frame buffer pool 2100 includes a number of individual frame buffers, such as frame buffers 2130, 2132, 2134, 2136, and 2138, that are used for various purposes. The free pool 2110 is organized as a ring buffer and is used to acquire and analyze frames for event determination. The write pointer 2120 indicates the next available frame buffer 2130 from which the next frame is obtained. Simultaneously with the image acquisition to the frame buffer 2130, the previous image in the frame buffer 2132 is analyzed. At some point, the ring buffer may fill up, at which point old frames are overwritten.

  In an exemplary embodiment where the imaging device 760 is an LM9630, for example, each frame buffer includes 128 × 100 8-bit pixels. In the figure, the frame buffer pool 2100 is shown to include only a few dozen elements for clarity, but in practice it is desirable to have more. In one embodiment, 160 elements are used, but storage slightly less than 2 megabytes is required, with an execution speed of 200 frames / second, approximately 0.8 seconds, or an execution speed of 500 frames / second. The ability to store in about 0.32 seconds. Obviously, the lower the frame rate, the more time an image can be stored.

  When an event is detected, many frames may occur since the event occurred, but the recent history of acquired images is in the free pool 2110. In the exemplary embodiment, free pool 2110 is large enough to hold event frames and frames before and after a sufficient number of event frames for application purposes. When an event is detected, the recent history may include some or all of the frames that are not recorded, depending on the user's choice, such as recording control 1050. If the recent history includes all of the frames, it can be recorded immediately, as explained below. If not all, recording will occur at a future time when all of the recorded frames are available.

  In an exemplary embodiment, a frame buffer marked “R”, such as Example 2134, holds the image to be recorded, and “E”, such as Example 2136, holds the event frame (recorded). The event occurs at recording time 2160 and is detected during analysis of the buffer 2134 frame. When an event is detected, some but not all of the recorded frames are in the recent history. During later analysis of the buffer 2132 frames, it is determined that the recent history now contains all the frames recorded.

  To record the frame, the frame buffer is deleted from the free pool 2110 and added to the stored event pool 2104, including stored events 2112, 2114 and 2116. If the number of frame buffers in the free pool 2110 becomes too small after deleting a newly stored event, various actions are possible. In one embodiment, event detection suspends until HMI 630 (FIG. 6) uploads the frame to stored event pool 2104 so that the buffer can return to free pool 2110. In another embodiment, one or more stored old events may be retrieved from the stored event pool 2104 and returned to the free pool 2104. These old events can no longer be displayed.

  In the exemplary embodiment, the frame buffer is never copied. Instead, the frame buffer moves between the free pool 2110 and the stored event pool 2104 by pointer manipulation using techniques well known in the art.

  A list of stored events 2102, such as list elements 2140, 2142 and 2144, is maintained. The list element 2140 includes, for example, a next element pointer 2150, a frame buffer count 2152, result information 2154, and a stored event pointer 2156. Result information 2154 may include a time stamp, as shown, or other information such as active data 302 that is not displayed.

  Result information 2154 includes information that applies to the entire event. It is further preferred to provide information for each recorded frame, as in the example displayed in the frame buffer of the stored event pool 2104. In the exemplary embodiment, the stored information includes a time stamp that records the acquisition time of the frame assumed for the recording time in milliseconds. Other information (not displayed) may also be recorded, such as event detection weights and individual device results.

  Referring back to FIG. 6, the vision detector may be connected to a human-machine interface (HMI) 630 via signal 640 for configuration purposes. It is also possible for the HMI to be part of the vision detector 600, but this is less preferred because the HMI is generally not needed for event detection. Thus, one HMI can be shared among multiple vision detectors. The HMI may implement a conventional design graphical user interface (GUI) of the illustrated portion shown in FIG.

  The GUI allows a portion of the recorded image stored in the vision detector memory 710 to be displayed to a human user. In the exemplary embodiment of FIG. 22, projection slide window 2202 displays up to eight thumbnail images 2210, 2212, 2214, 2216, 2220, 2230, 2232 and 2234. Each thumbnail image is a low resolution version of the recorded image corresponding to the stored event pool 2104. In general, a thumbnail image corresponds to a continuous image of recorded single events, but other processes may be useful, such as omitting some of the images between corresponding thumbnails.

  A set of scroll controls 2250 is provided to the projection slide window 2202 for moving thumbnail images back and forth within the recorded images of events or between events. The next image control 2260 advances the image one by one, and the previous image control 2262 advances the image one by one in reverse. The next event control 2264 and the previous event control 2266 display events one after the other.

  Thumbnail 2220 displays a low-resolution image of object 2242 that may correspond to the example of object 116 (FIG. 1). The object 2242 also displays all other thumbnails at slightly different display viewpoints (positions within the field of view) and at different times during the application of the label 2270 by the arm 2272, as in the example object 2240 of the thumbnail 2210. . By issuing scroll commands using the scroll control 2250, the user can move the recorded image back and forth to view any desired time interval. In particular, considering the illustrative images in thumbnails 2212, 2214, and 2216, label 2270 appears to be stuck at the top of the object as it is pasted.

  In the exemplary embodiment of FIG. 22, the image corresponding to the thumbnail 2220 that is surrounded by a thick line and referred to as the “selected image” is also displayed in the image display window 2200 at the highest resolution. As the scroll command advances forward and backward through the display portion, another selected image moves to thumbnail 2220 and is displayed at the highest resolution in image display window 2200. Other information about the selected image is also displayed, such as a time stamp 2280 indicating the acquisition time (milliseconds in this example) of the image that was supposedly selected for the recording time.

  The above is a detailed description of various embodiments of the invention. It will be apparent that various modifications and additions can be made to the invention without departing from the product and scope of the invention. For example, the processors and computing devices herein are examples and various processors and computers, both stand-alone and distributed, can be employed to perform the disclosed calculations. Similarly, the imaging devices and other vision components described herein are examples, and improved or alternative components can be employed within the present disclosure. Software elements, GUI design and layout, parameter values, and mathematical expressions can all be equally modified or replaced to suit the particular application of the present invention. Accordingly, this description is meant to be exemplary, and is not intended to limit the scope of the present invention.

Claims (8)

A method for detecting and reporting a recording time,
Generating a relative motion between a set of objects including at least one object and a two-dimensional field of view of a sensor having a global shutter;
Providing a recording point at a fixed position relative to the sensor;
Acquiring a plurality of active frames of the two-dimensional field of view using the sensor at a plurality of acquisition times;
Performing an analysis of the plurality of active frames to obtain a plurality of location values, wherein one location value of the plurality of location values is the one at the acquisition time at which the corresponding active frame was acquired; Performing an analysis of the plurality of active frames corresponding to a position of one object of the set with respect to the recording point to obtain a plurality of location values;
Determining a corresponding recording time for the intersection of at least a portion of the recording points by one object of the set of objects using the acquisition time and a location value; A recording time determination is made at a corresponding decision point that differs from the corresponding recording time by a corresponding decision delay, and the decision delay is substantially variable, according to one object of the set of objects. Determining a corresponding recording time using the acquisition time and a location value for at least some intersections of the recording points;
Generating a signal at a report time after a delay time interval from the corresponding recording time for at least a portion of the recording time;
Using the reporting time of the signal to indicate the corresponding recording time.   The method of claim 1, wherein determining the recording time further comprises fitting a curve to at least three acquisition times and corresponding location values.   The method of claim 2, wherein the curve comprises a line.   The method of claim 1, further comprising using a FIFO buffer to hold information necessary to generate the signal at the report time. A system for detecting and reporting recording time,
Means for generating relative motion between a set of objects including at least one object and a two-dimensional field of view of a sensor having a global shutter;
Means for providing a recording point at a fixed position relative to the sensor;
Means for acquiring a plurality of active frames of the two-dimensional field of view using the sensor at a plurality of acquisition times;
Means for performing an analysis of the plurality of active frames to obtain a plurality of location values, wherein one location value of the plurality of location values is the one at the acquisition time at which the corresponding active frame was acquired; Means for performing an analysis of the plurality of active frames corresponding to a position of one object of the set relative to the recording point to obtain a plurality of location values;
Means for determining a corresponding recording time using the acquisition time and a location value for an intersection of at least some of the recording points by one object of the set of objects, A recording time determination is made at a corresponding decision point that differs from the corresponding recording time by a corresponding decision delay, and the decision delay is substantially variable, according to one object of the set of objects. Means for determining a corresponding recording time using the acquisition time and location value for at least some intersections of the recording points;
Means for generating a signal at a report time after a delay time interval from the corresponding recording time for at least a portion of the recording time;
Using said reporting time of the signal, the system comprising a means for indicating the recording time said corresponding.   6. The system of claim 5, wherein the means for determining the recording time further comprises means for fitting a curve to at least three acquisition times and corresponding location values.   The system of claim 6, wherein the curve comprises a line.   6. The system of claim 5, further comprising means for holding information necessary to generate the signal at the report time using a FIFO buffer.

Images