TWI759534B - Activity recognition system based on machine vision for multiple objects - Google Patents

Abstract

This patent proposes an activity recognition system which can detect not only action of a single object but activity in the scene by machine vision. Vision information is input by vision sensor(s) and passes to object recognition unit to produce a data structure of classification of objects and spatial relationship amount them. Pose estimation unit detects posture of each object in vision information. All semantic information is collected and integrated in fusion analysis unit and then activity recognition is finished.

Description

Visual activity recognition system based on multiple objects

The present invention relates to a system for activity recognition. In particular, a system based on vision for activity recognition.

Vision-based activity recognition systems use visual information to identify possible activities in visual information. Activity is not the action of a single object in a certain frame, but refers to the activity (Activity) composed of one or more objects, the individual or the action relationship between them; such as a single object, Taking a person as an example, its action is identified as running; however, if multi-object detection is performed, if a group of people is detected to be running, their activity can be identified as a marathon; if the background object of multi-object detection is outside the prison wall, the The activity can be identified as a prison escape; if the detective holds a gun and another object is followed by a police officer running and chasing, the activity can be identified as chasing a fugitive. The prior art mostly recognizes a single action of a specific tracking object, rather than activity detection that detects the relationship between the scene and each object. If it is only the action recognition of the prior art, the recognition results are limited in terms of industrial utilization benefits. In addition, the prior art's focus on tracking a specific object within a range of visual data also limits the ability to achieve activity-level discrimination. In addition, the prior art is mostly based on time-sequenced dynamic images. After the time-sequenced dynamic images are cut into image sequences, the before and after images are compared, and the relative relationship between the relevant features of the tracked object is locked to perform motion recognition. The calculation of the tracked object is: Feature tracking and comparison between multiple image sequences based on time series often consumes a lot of computation in addition to the motion recognition operation itself, and it is difficult to have the ability to track a large number of objects with limited computing resources, resulting in performance bottlenecks and high resource thresholds.

The present invention proposes an activity recognition system, but it can also be applied to motion recognition, which can generate a comprehensive semantic description of the relationship between each object in the scene and each other according to the image information. Using this comprehensive semantic description can generate not only actions, but also It is the recognition result of the activity level, which has greatly improved the value and application of machine vision in the industry. And its fusion analysis process is the statistics of the static image recognition result set, one does not necessarily need to use dynamic images, and the other does not require additional object feature tracking between time-series image sequences, which can greatly improve the efficiency of recognition, and at the same time can It has the performance of recognizing a large number of objects in the image, and achieves the overall effect of more efficiency, lower cost and higher function. The present invention is a key development for future machine intelligence to recognize complex human activities and take corresponding feedback and actions, and has high industrial value.

After the visual sensing device (such as a camera) receives the image information (which can be a single static image), the image information is sent to the object recognition unit, and the object recognition unit uses relevant algorithms (such as machine learning, or arithmetic based on rules) to interpret The types of objects in the screen and the positions of the objects in the screen are output. After processing, the corresponding relationship between the objects can be generated (such as information such as front, back, left, right, up, down, and relative distance ratios, etc.), so that the image information can be obtained. scene information. On the other hand, the gesture recognition unit can receive the original image information of the visual sensing device, and/or part of the object image information after interpretation by the object recognition unit, so as to recognize the gesture information of each object. Algorithms (such as machine learning, or operations based on rules), generate the posture interpretation of the human body, such as standing, lying, sitting, and lying down; in the case of vehicles, it can be posture information such as door opening, window rolling down, etc.; posture recognition unit After the operation, the posture state of each object in the visual information is generated. The object type information and/or spatial relative relationship information, etc. generated by the object recognition unit, plus the status information of each object generated by the gesture recognition unit, the fusion analysis unit can integrate the data, the conditions of each object and the relationship between each other. The relative relationship will be clear, this is the scene information, which is the semantic data, and then the fusion analysis unit performs the operation of the identification algorithm (such as machine learning, or the operation based on the rules) on the scene information, and generates an activity interpretation result. As a result, valuable various applications can be made. The system thus has the ability to discriminate the entire scene, rather than a single action of a single object, resulting in activity-level interpretations. After the activity interpretation result is generated, the system can also have an event judging unit, which is set to generate a specific event notification when a related activity and/or a certain action occurs. The processing of actions, such as issuing an alarm, sending out a warning message, activating the relevant actuator or device, etc. In addition, this system does not necessarily need a series of dynamic images, and can use a single static image information for identification, but if there are dynamic images, it can also be used to increase the accuracy of activity interpretation, but does not require tracking and comparison operations, fusion analysis The unit adds the object type information, and/or spatial relative relationship information, etc. generated from the time-series-based image sequence, plus the state information of each object, that is, the scene information, and puts it into the relevant algorithm once or in stages according to each image sequence. In the middle interpretation, the activity interpretation results of each image sequence are counted, and there is no need to track the object feature position for each image sequence. If the interpretation results of each image sequence are simply counted, the most suitable activity interpretation results can be output. The image is divided into several static images, and after the scene information generated by several static images is interpreted by the fusion analysis unit, the same or different number of activity interpretation results will be generated. For example, the simple law of large numbers can be used, the more is the winner, and 100 event interpretation results are generated. If 90 are marathons and 10 are parades, the result is the marathon). This system will reduce the computational cost of object tracking and reserve computational resources for the identification calculation itself. In addition, this system can also be combined with other visual sensors (such as infrared cameras, depth cameras, etc.), or non-visual sensors (such as gyroscopes installed on objects, etc.) It is brought into the fusion analysis unit, and the fusion analysis unit adds additional related algorithms for operation. In addition, the fusion analysis unit can also be an independent server and/or cloud system, and the relevant scene data can be connected to each other by wired network, wireless network, or other related transmission methods such as radio waves and light waves. More efficient or secure computing. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since those skilled in the art will understand the scope of the invention from this detailed description various changes and modifications. Therefore, it is to be understood that the invention is not limited to the particular component parts of the devices described or the steps of the methods described, as such devices and methods may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It must be noted that, as used in this specification and the appended claims, the articles "a" and "the" are intended to mean that there are one or more of the elements unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" can include several devices, etc. Furthermore, the words "comprising", "comprising", "comprising" and similar words do not exclude other elements or steps.

Based on the above, the present invention is based on a visual activity recognition system based on multiple objects, which can identify the type of objects and the relative relationship in space, identify and integrate the gestures of each object, generate semantic scene information, and then perform activity level recognition to achieve machine intelligence. The ability to take corresponding feedback and action based on the recognition of more complex human activities is a very critical development. Its characteristics not only increase the technology of activity recognition in the existing industry, but also expand the application of existing image recognition technology, such as detection, exploration, security, production monitoring, home medical treatment, etc. It is highly industrially applicable. , not only can be operated by using static images, but also can generate results using a simple non-tracking statistical method in moving images, which has not yet been achieved in the prior art, which is novel and progressive. In conclusion, this case is indeed an invention with novelty, progress, and industrial application value, and has met the requirements of an invention patent, so a patent application was filed in accordance with the law. However, the above are only the preferred embodiments of the present invention, and cannot limit the scope of the patent application of the present invention. Any equivalent changes or modifications implemented in accordance with the spirit of the present invention should still be covered by the patent of the present invention. within the range.

The present invention will now be described more fully with reference to the accompanying drawings, in which presently preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; these embodiments are provided for thoroughness and completeness, and fully convey the scope of the invention to those skilled in the art . FIG. 1 illustrates a visual activity recognition system 10 that can be based on multiple objects. The system includes a digital network camera 110, and its image data is transmitted to the object recognition unit 120 through a transmission interface using a wired network or a wireless network or Bus bar or other light wave connection. The object recognition unit 120 recognizes the objects in the image information, and generates an object recognition data structure as shown in FIG. 2 , which is a graph structure in graph theory, including object types and spatial positions as nodes, and through edges (edge) are connected to each other, the edge has its characteristic representation relationship (such as front, back, left, right, up, down), and can carry distance information (calculated based on the proportion of the static picture, such as 1% means that the distance between two objects is % of the picture) 1), the data structure can be transmitted to the fusion analysis unit 140 together with the image information, and the transmission interface is the connection method between the digital network camera 110 and the object recognition unit 120 described above. The gesture recognition unit 130 receives the image information from the digital network camera 110 and/or the object recognition data structure from the object recognition unit 120, and then generates the gesture information of each object after calculation, and transmits the gesture information of each object to the transmission interface. The fusion analysis unit 140, after the fusion analysis unit 140 integrates the data structure of FIG. 2 and the posture information of each object, it generates scene information, which is a semantic type rather than an image type data, and the scene information can use the relevant algorithm (such as seq2seq, RNN and other semantic type algorithms) to identify and obtain the result of activity identification. The result is transmitted to the event judging unit 20 through the transmission interface. The event judging unit has a rule database. If the action recognition result meets the conditions for triggering the event in the rule database, the event judging unit 20 can start the corresponding action.

10‧‧‧Visual Activity Recognition System 110‧‧‧Visual Sensing Equipment (such as Digital Network Camera) 120‧‧‧Object Recognition Unit 130‧‧‧Gestation Recognition Unit 140‧‧‧Fusion Analysis Unit 20‧‧‧Event Judgment unit

[Fig. 1] is an architecture diagram of a visual activity recognition system according to the present invention. [Fig. 2] is an example diagram of the structure of the object types and spatial relationship data generated by the object recognition unit of the present invention.

10‧‧‧Visual Activity Recognition System

110‧‧‧Visual sensing equipment (such as digital network cameras)

120‧‧‧Object recognition unit

130‧‧‧ Gesture Recognition Unit

140‧‧‧Fusion Analysis Unit

20‧‧‧Event Judgment Unit

Claims (8)

A visual activity recognition system that can be based on multiple objects at least includes: at least one visual sensing device; an object recognition unit, which is connected to the visual sensing device, and is used to include in visual information according to the data provided by it. Object identification is performed on one or more objects to generate an object identification data structure, and the object identification data structure includes one or more object types and/or spatial information and/or spatial relationships between objects in the visual information, wherein The spatial relationship between the objects can be obtained by calculating the image information in proportion; a gesture recognition unit is connected with the visual sensing device and/or the object recognition unit, and is used for recognizing the visual action from the visual action recognition device and/or the object recognition unit. Or the visual information output by the object recognition unit to perform gesture discrimination including one or more objects; a fusion analysis unit, which is connected with the gesture recognition unit and/or the object recognition unit, is used to determine the type and type of one or more objects and After all or part of the information of the gesture discrimination data and/or the spatial information and/or the spatial relationship between objects is integrated, a scene information is generated, and the scene information is a semantic type rather than an image type data, and the scene information is Recognition can be performed using a semantic algorithm to generate activity and/or action discrimination results for the visual data. The visual activity recognition system according to claim 1 is further connected to an event judging unit, and when the activity and/or the action result meets the set conditions, a corresponding processing action is triggered. According to the visual activity recognition system described in claim 2, the event judgment unit sets one or more conditions, and the objects corresponding to the conditions are also one or more. The visual activity recognition system of claim 1, wherein the visual sensing device is a dynamic image sensing device. The visual activity recognition system according to claim 4, wherein the fusion analysis unit calculates and analyzes the results output by the object recognition unit and/or the gesture recognition unit of a certain number of still images segmented based on time series. The visual activity recognition system according to claim 5, wherein the calculation is based on the total number of samples and the result number of each sample activity for statistical calculation, so as to generate the final activity recognition result. The visual activity recognition system according to claim 1 is connected with other visual and/or non-visual type sensors, and is analyzed together by a fusion analysis unit. The visual activity recognition system according to claim 1, wherein the fusion analysis unit is an independent server and/or a cloud computing environment, and is connected by means of network, radio wave, or other carrier transmission.

Images