JP2023546189A - Classification device, control device, classification method, control method and program - Google Patents

Abstract

The purpose of the present disclosure is to provide a classification device that can provide assistance to humans. The classification device includes a generating means (11) for determining a specific time region of video data based on a predetermined algorithm and generating partial video data from which video data is extracted in the specific time region; and a modification means (13) for modifying a predetermined algorithm based on the evaluation of the classification performed by the classification means. [Selection diagram] Figure 1

Description

The present disclosure relates to a classification device, a control device, a classification method, a control method, and a non-transitory computer-readable medium.

Image and video analysis techniques have developed rapidly.

For example, Patent Document 1 discloses a display control device that can create thumbnails of scenes. Specifically, the display control device creates a clustering result in which each frame of the content is subject to clustering, and displays thumbnails. The scene classification unit 612 of the display control device classifies frames belonging to the cluster of interest into a scene having a frame group consisting of one or more frames. The thumbnail creation unit 613 of the display control device creates a thumbnail for each scene based on the scene information from the scene classification unit 612.

Patent No. 5533861

In recent years, technology has been developed to support human activities with machines (eg, computers, support robots, etc.). In such technologies, it is important for machines to detect and classify human movement sequences in order to provide the assistance desired by humans.

The aim of the present disclosure is to provide a classification device, a control device, a classification method, a control method and a non-transitory computer-readable medium capable of providing human assistance (understood as assistance to humans).

In a first exemplary aspect, the classification device includes a generating means for determining a specific temporal region of video data based on a predetermined algorithm, and generating partial video data from which the video data is extracted in the specific temporal region. and a classification means for classifying the partial video data generated by the generation means, and a modification means for modifying the predetermined algorithm based on the evaluation of the classification performed by the classification means.

In a second exemplary aspect, the control device comprises: recognition means for recognizing video data including a task and thereby determining said task; and determining an operation of the machine in response to said determined task; and a controller for controlling the machine according to the selected task.

In a third exemplary aspect, the classification method includes determining a specific temporal region of video data based on a predetermined algorithm, and generating partial video data from which the video data is extracted in the specific temporal region. , classifying the partial video data and modifying the predetermined algorithm based on the classification evaluation.

In a fourth exemplary aspect, the control method includes: recognizing video data including a task, thereby determining the task; and determining an operation of a machine in response to the determined task; controlling the machine according to the task.

A fifth exemplary aspect includes determining a specific time region of video data based on a predetermined algorithm, and generating partial video data from which the video data is extracted in the specific time region; A non-transitory computer-readable medium that stores a program that causes a computer to classify data and modify the predetermined algorithm based on an evaluation of the classification.

A sixth exemplary aspect includes recognizing video data including a task, thereby determining the task, determining an operation of a machine in accordance with the determined task, and determining an operation of a machine in accordance with the determined task. A non-transitory computer-readable medium that stores a program that causes a computer to control and execute a computer.

According to the present disclosure, it is possible to provide a classification device, a control device, a classification method, a control method, and a non-transitory computer-readable medium that can provide assistance to humans.

FIG. 1 is a block diagram of a classification device according to a first embodiment. FIG. 2 is a flowchart showing a video data classification method according to the first embodiment. FIG. 3 is a block diagram of a control device according to a second embodiment. FIG. 4 is a flowchart showing a method for controlling a machine according to the second embodiment. FIG. 5 is a block diagram of a classification system according to the third embodiment. FIG. 6 is a block diagram of a generation unit according to the third embodiment. FIG. 7 is a graph showing an example of an intensity signal of video data according to the third embodiment. FIG. 8A is a diagram illustrating an example of human motion in each subsequence according to the third embodiment. FIG. 8B is a table showing examples of classifications and category labels corresponding to subsequences according to the third embodiment. FIG. 9 is a graph showing an example of an intensity signal of video data according to the third embodiment. FIG. 10 is a table showing examples of classifications and category labels corresponding to subsequences according to the third embodiment. FIG. 11 is a schematic diagram of feedback processing according to the third embodiment. FIG. 12 is a graph showing an example of changes in the number of rational classification solutions according to the third embodiment. FIG. 13 is a block diagram of an intention detection system according to the fourth embodiment. FIG. 14 is a block diagram of a machine including the intention detection system according to the fifth embodiment. FIG. 15 is a diagram illustrating an example of a picking robot including the intention detection system according to the fifth embodiment. FIG. 16A is a diagram illustrating an example of processing of a picking robot instructed by a human gesture according to the fifth embodiment. FIG. 16B is a diagram illustrating another example of processing of the picking robot instructed by a human gesture according to the fifth embodiment. FIG. 17 is a configuration diagram of an information processing device according to an embodiment.

(Embodiment 1)
Embodiment 1 of the present disclosure will be described below with reference to the drawings. Referring to FIG. 1, the classification device 10 includes a generation section 11, a classification section 12, and a modification section 13. Classification device 10 may be applied to various computers or machines capable of handling video data. For example, the classification device 10 may be installed as a personal computer, video recorder, robot, machine, television, mobile phone, etc.

The generation unit 11 determines a specific time region of video data based on a predetermined algorithm, and generates partial video data from which video data is extracted in the specific time region. Video data has a certain time length, and a particular time region is within a certain time length. The video data may be a sequence of image data. That is, video data may have multiple frames. The generation unit 11 may analyze the content of the video data using a predetermined algorithm and set a specific time domain. The video data may be stored in a memory within the classification device 10, or may be input to the generation unit 11 from outside the classification device 10. Furthermore, the predetermined algorithm may be stored in memory within the classification device 10.

The classification unit 12 classifies the partial video data generated by the generation unit 11. Classification can be performed using numbers, text, or the like. Classifications may relate to human actions such as, but are not limited to, gestures, specific scenes from television shows or movies.

The modification unit 13 modifies a predetermined algorithm based on the classification evaluation performed by the classification unit 12. The evaluation may be processed by a component within the classification device 10, or may be processed by a device outside the classification device 10.

FIG. 2 is a flowchart illustrating an example of a process executed by the classification device 10 according to the first embodiment. The processing executed by the classification device 10 will be described below.

First, the generation unit 11 determines a specific time region of video data based on a predetermined algorithm (step S11). Next, the generation unit 11 generates partial video data in which video data is extracted in a specific time domain (step S12). This partial video data may point to a scene and show a type of human action, but is not limited thereto.

Next, the classification unit 12 classifies the partial video data generated by the generation unit 11 (step S13). Through this process, the classification unit 12 may classify various partial video data into a plurality of categories.

Thereafter, the modification unit 13 modifies the predetermined algorithm as necessary based on the evaluation of the classification performed by the classification unit 12 (step S14). As a result of modifying the predetermined algorithm, the specific time domain may be changed along with the evaluation results. Therefore, by generating partial video data, the generation unit 11 can include scenes to be classified more accurately in the partial video data. For example, if the classification is aimed at classifying human motion subsequences, the classifier 10 may determine appropriate temporal regions of partial video data representing one human motion subsequence. . As a result, the classifier 12 can more accurately classify the partial video data because the partial video data can represent an accurate motion subsequence of one person.

(Embodiment 2)
Embodiment 2 of the present disclosure will be described below with reference to the drawings. Referring to FIG. 3, the control device 14 includes a recognition unit 15 and a controller 16. The control device 14 may be applied to devices mounted on various computers or machines, such as robots that assist humans.

The recognition unit 15 recognizes video data including an operation, and determines the operation accordingly. The video data may show human motion, and the human motion may be a task on an object. For example, the work includes the work of grasping a certain object, the work of placing a certain object, and the like. This gesture can instruct the robot to perform some processing, and can be implicit or explicit. Video data may be classified as shown in Embodiment 1.

The controller 16 determines the operation of the machine according to the determined work, and controls the machine according to the determined work. The machine may include, but is not limited to, the controller 14.

FIG. 4 is a flowchart illustrating an example of processing executed by the control device 14 according to the second embodiment. The processing executed by the control device 14 will be described below.

First, the recognition unit 15 recognizes video data including work (step S15). As mentioned above, the work may be a human action. Next, the recognition unit 15 determines the work by recognizing the video data (step S16).

Then, the controller 16 determines the operation of the machine according to the determined work (step S17). After that, the controller 16 controls the machine according to the determined work (step S18). For example, when a user performs a task, the recognition unit 15 understands what the user wants the machine to do, and the controller 16 can control the machine as directed by the user or other input. Specifically, through this process, the control device 14 can control the machine by recognizing the human intention.

The control device 14 according to the second embodiment can reduce system integration functions in machines such as robots and computers, for example.

The recognition unit 15 can be realized by the functions of the classification unit 12 and/or correction unit 13 in FIG. Furthermore, the recognition unit 15 may be realized by the functions of the preprocessing unit 21, the generation unit 22, the classification unit 23, the mapping unit 24, and/or the modification unit 25 shown in FIG. The recognition unit 15 may be realized by the functions of the calculation unit 26, the signal analysis unit 27, the determination unit 28, and/or the subsequence generation unit 29 in FIG. Furthermore, the recognition unit 15 may be realized by the functions of the human object analysis unit 31 and/or the intention detection unit 32 shown in FIG. The recognition unit 15 may be realized by a pattern recognition algorithm and/or an image recognition algorithm in the field of computer vision. Furthermore, the controller 16 can be realized by the functions of the signal generator 41 and/or the optimizer controller 42 of FIG. Details of FIGS. 5, 6, 13 and 14 will be described later.

(Embodiment 3)
Embodiment 3 of the present disclosure will be described below with reference to the drawings. Embodiment 3 is a specific example of Embodiment 1.

First, the configuration and processing of the classification system according to the third embodiment will be explained. Referring to FIG. 5, the classification system 20 includes a preprocessing section 21, a generation section 22, a classification section 23, a mapping section 24, a modification section 25, and a database (DB). Classification system 20 may be provided as a mechanical or robotic module, for example. Classification system 20 may receive raw video data from a sensory input or an imaging section (not shown in FIG. 5), such as a video camera. The imaging section can capture frames of a person at regular intervals.

The preprocessing unit 21 receives raw video data and preprocesses it (ie, preprocesses it). Specifically, the preprocessing unit 21 reduces information included in the raw data and generates preprocessed video data (hereinafter simply referred to as video data) that includes information regarding classification. This is done by the classification section 23. For example, the preprocessor 21 may reduce a sequence of irregularly sampled high resolution frames to frames with a small number of data points containing relevant information. The related information may include characteristic body points of the person being photographed. Further, the related information may include a relationship between a person and an object that the person works on or is located near the person.

The preprocessing unit 21 outputs video data to the generation unit 22. The preprocessing unit 21 may be realized by a combination of preprocessing software and a processor in the classification system 20.

The generation unit 22 receives the video data from the preprocessing unit 21 and generates a subsequence (partial video data) from which video data is extracted in a specific time domain. To this end, the generation unit 22 determines a specific time region of the video data based on a predetermined algorithm. That is, the generation unit 22 may be implemented as a division generation unit that divides the video into a plurality of subsequences.

FIG. 6 is a block diagram of the generation unit 22. The generation unit 22 includes a calculation unit 26, a signal analysis unit 27, a determination unit 28, and a subsequence generation unit 29. Detailed processing in the generation unit 22 will be explained.

The calculation unit 26 calculates an intensity signal of the video data to determine the length of the subsequence and the position of the subsequence (ie, a specific time domain) within the video data, where the intensity signal is indicative of human motion. In particular, by means of a scalar signal and by determining feature points of this signal, the intensity signal summarizes the dynamic movements of the moving person. The calculation unit 26 calculates the intensity signal using a predetermined algorithm that may be expressed as a formula and/or a rule. The calculation unit 26 outputs the intensity signal to the signal analysis unit 27.

The signal analysis unit 27 analyzes the intensity signal and identifies intensity candidate points. The candidate points are feature point candidates for determining the length of the subsequence and the position of the subsequence in the video data.

The determining unit 28 determines feature points from the candidate points identified by the signal analyzing unit 27. The signal analysis unit 27 and the determination unit 28 perform the above processing using a rule base included in a predetermined algorithm. In this way, feature points in the video data are derived.

The subsequence generation unit 29 uses the feature points determined by the determination unit 28 to determine the length of the subsequence and the position of the subsequence in the video data. The subsequence generation unit 29 determines these elements using generation rules included in a predetermined algorithm. The subsequence generator 29 generates a subsequence from a sequence of frames (ie, video data).

In summary, the generator 22 can generate a series of subsequences from a sequence of frames of video data based on a predetermined algorithm that includes generative laws and a suitable rule base. The data of each subsequence is used by the classification unit 23 as a motion subsequence candidate. The generation unit 22 outputs the generated subsequence to the classification unit 23.

It should also be noted that the predetermined algorithm can be modified by feedback from the modification unit 25. If the predetermined algorithm is modified, the method by which the calculation unit 26 calculates the intensity signal is changed and/or the method by which at least one of the signal analysis unit 27 and the determination unit 28 determines the feature points is changed. Therefore, the length of the subsequence and/or the position of the subsequence within the video data is modified to obtain a more accurate classification. This correction process will be explained in detail below.

The classification unit 23 receives the subsequence and classifies the subsequence (partial video data) as a human motion. The classification unit 23 assigns the classified subsequences to classification numbers. Furthermore, the classification unit 23 can assign the classified subsequences to text labels by accessing the mapping unit 24 and/or the DB. Subsequences are classified as clusters of human actions. The classification unit 23 outputs the subsequences along with classification numbers and text labels for further processing.

Furthermore, the classifier 23 may derive one (or temporarily more than one) classification solution in order to consider a single subsequence as a candidate to be classified. The classifier 23 performs this process for each subsequence, and a classification solution is needed for the classifier 23 to classify the subsequences.

The DB functions as a library and stores classification solutions and classification numbers generated by the classification unit 23. The classification unit 23 can access the DB and classify subsequences using the classification solution and classification number.

The mapping unit 24 acquires text information related to the classification of documents and the like from a database and/or the Internet. The mapping unit 24 further obtains textual information provided by users of the classification system 20, among others. The mapping unit 24 processes the text information and generates a mapping of vocabulary to descriptions used for classification. The mapping unit 24 may include an input unit and/or a network interface in addition to a processor and a memory. The classification unit 23 can access the mapping unit 24 and refer to the mapping in order to improve the accuracy of subsequence and category determination. In other words, the classification process performed by the classification unit 23 is supported by mapping categories to linguistic domains generated by the mapping unit 24. More specifically, the classification unit 23 assigns human-understandable text labels to subsequences, and assigns motion patterns as accurately as possible to already identified categories that have been labeled with classification numbers. Describe. It is also useful in cases where the classifier 23 cannot determine the category using the text labels of adjacent subsequences. The main purpose of using textual information is to enhance the classification capabilities and to understand the system's reasoning in case erroneous results of the categorizer require additional tuning or correction.

The modification unit 25 determines the evaluation value of a specific classification solution. The evaluation value of a classification solution may indicate how suitable the corresponding classification is for subsequent processing steps after classification. An example of a subsequent processing step is intent detection. The evaluation value may indicate how well the corresponding classification is suitable for predicting subsequent actions or events of the human movement indicated by the corresponding subsequence.

The evaluation value of the obtained classification solution can be determined based on one or more indicators. The first example of an indicator is how well the classification solution (i.e., the classification unit 23) classifies elements that are already known to belong to the same category as part of the same category. be. A second example of an indicator is an indicator that describes the deviation from a predetermined number of categories for a defined problem. In other words, this metric indicates how much there is a deviation from the optimal number of categories assumed to be known for the defined problem. For example, the more inappropriate the classification solution obtained, the larger this indicator becomes. A third example of a metric is a metric that describes how well the entire system, including the classification system 20, accomplishes the overall task using the classification solution. This is one of the most important metrics and the most difficult one if it should be used to improve classification. An example of the entire system will be described later.

The modification unit 25 evaluates the classification using at least one of these indicators. However, indicators are not limited to these examples. The indicator may have various parameters to define the correctness or suitability of the classification solution. If the evaluation value of the current classification solution does not satisfy a predetermined criterion regarding the index (for example, if the current classification solution is not sufficient for the task under consideration), the modification unit 25 instructs the generation unit 22 in a predetermined algorithm. Give appropriate instructions (feedback) to make changes. Specifically, if the modification unit 25 considers the classification evaluation and determines that a certain category is not appropriate, the modification unit 25 determines that the part of the predetermined algorithm that corresponds to that category should be modified. Send instructions to instruct. The predetermined algorithm is modified based on the instruction, and at least one of the calculation method by the calculation unit 26, the analysis method by the signal analysis unit 27, the determination method by the determination unit 28, and the generation method by the subsequence generation unit 29 is modified. Added. As a result, the length of the subsequences and the position of the subsequences in the video data can be changed.

Next, with reference to FIGS. 7 to 8B, specific examples of human movements and processing executed by the classification system 20 will be described. An example of the intensity signal of video data is shown in FIG. FIG. 7 shows the time axis from frame number 0 to _kc , and there are two feature points of the frame, _kA and _kB . As shown in FIG. 7, at the feature point, the signal strength has an inflection point, especially a minimum value, with respect to the strength _dk .

In the classification system 20, the calculation unit 26 derives the graph of FIG. The signal analysis unit 27 analyzes this graph, finds two feature points _kA and _kB , and sets these two points as candidate points. Then, the determining unit 28 sets the two points _kA and _kB as feature points. The subsequence generator 29 uses the determined two points _kA and _kB to determine the length and position of the subsequence in the video data. In this example, the subsequence generation unit 29 generates subsequences (1), (2), and (3). Subsequence (1) is the subsequence from frame number 0 to _kA , subsequence (2) is the subsequence from frame number _kA to _kB , and subsequence (2) is the subsequence from frame number _kB to _kc . 3). As mentioned above, a subsequence is defined by two feature points k _A and k _B.

FIG. 8A shows an example of human motion for each subsequence. As shown in FIG. 8A, subsequence (1) is for person P to “raise his left arm,” subsequence (2) is for person P to “pass the object” regarding object O, and subsequence (3) is for person P to “raise his left arm.” P indicates "relaxation". These characteristic body points of human motion are represented by the intensity signals in FIG.

FIG. 8B shows examples of categories and category labels corresponding to subsequences (1) to (3). The category of subsequence (1) is "mp31," the category of subsequence (2) is "mp76," and the category of subsequence (1) is "mp21." The classification unit 23 uses the DB to set these category numbers. Furthermore, the category label of subsequence (1) is "raise left arm," the category of subsequence (2) is "pass the object," and the category of subsequence (1) is "relax." The classification unit 23 uses the text information generated by the mapping unit 24 to set these category labels. In this manner, classification system 20 defines labels for subsequences.

Next, an example in which the classification system 20 does not classify subsequences will be described with reference to FIGS. 9 and 10. The graph in FIG. 9 is the same as the graph in FIG. However, since there is a lack of information to help find the feature points, the classification system 20 incorrectly judges the false points k _A ′ and k _B ′ to be feature points. As a result, the subsequence generation unit 29 generates subsequences (1)', (2)', and (3)'. The subsequence from frame number 0 to _kA ' is subsequence (1)', the subsequence from frame number _kA ' to _kB ' is subsequence (2)', and the subsequence from frame number _kB ' to _kc is Set the subsequence as subsequence (3)'.

FIG. 10 shows examples of categories and category labels corresponding to subsequences (1)' to (3)'. The classification unit 23 can correctly determine the category and category label of subsequences (1)' and (3)', but cannot determine the category of subsequence (2)'. cannot be classified by In this case, text inference supports the classification process and allows classification even when classification would not be possible without this means.

FIG. 11 is a schematic diagram of the feedback processing by the correction unit 25 in this situation. The modification unit 25 uses the index to evaluate the classification result and sends feedback to the generation unit 22. The feedback indicates that a given algorithm needs to be modified with respect to feature point determination. In response to the feedback, the generator 22 uses a feature point re-evaluation algorithm and adjusts the feature point determination as a result of the re-evaluation. Thereby, the generation unit 22 moves the point k _A ′ and the point k _B ′ from the original points, as shown in FIG. 9 , and sets the points at the correct positions in FIG. 7 .

For example, the modification unit 25 may process a feature space including the pair of features 1 and 2 and classify the data points into groups in different ways. However, it should be noted that the feature space is not limited to two dimensions.

FIG. 12 shows an example of the evolution of the number of reasonable classification solutions. The classification unit 23 determines what kind of classification solution is a reasonable classification solution. At the start of FIG. 12, the number of reasonable classification solutions is one. The number becomes 2, 3, 2, 1, 2, 1 in order as time passes. In summary, the number may temporarily be multiple, but it converges to 1 over time. In order to perform post-classification processing, ambiguity should be reduced during classification, so it is desirable for the classification unit 23 to limit the number of classification solutions used by the classification unit 23 to one.

To detect human motion, related techniques may involve determining a category of human motion. However, if you are working on automatically determining categories using preprocessed data obtained by analyzing frames of a movie showing human actions (e.g., performing a specific task), then Problems may occur.

The first problem is that of deriving meaningful categories that describe subsequences using minimal or no information. The categories should make sense from a practical point of view, meaning that the resulting classification is useful for the overall purpose of the technical system. This problem occurs even if the correct length of the subsequence is precisely known. There is a need to establish valid criteria to describe the evaluation value of classification solutions.

The second problem is that of finding valid subsequences that can be mapped to categories and improving the subsequence decisions over time. This problem arises because it is difficult to derive the length of a single subsequence using only preprocessed data when there is no information or a small amount of information. Furthermore, there is no instruction to generate subsequences.

A third problem is that of using textual information, such as databases, documents obtained from the Internet, or especially provided by users, to improve the determination of subsequences and categories.

Classification system 20 can solve the aforementioned problems. The first problem is solved by setting an index for evaluating the classification performed by the classification unit 23. In related art, when the amount of relevant information available is small and the classification system provides input to further processing systems (such as an intention detection system), the impact cannot be directly estimated, i.e., when a behavior pattern classification system There are inherent difficulties in evaluation regarding how well local intent can be detected. However, with the introduction of an expanded set of indicators, the classification system 20 is able to evaluate the characteristics of the classification.

Furthermore, based on the classification evaluation, the classification system 20 can modify the predetermined algorithm to change the subsequence generation method (candidate point selection method), if necessary. That is, based on the evaluation value of the obtained classification solution, the calculation of the subsequence length may be adapted, for example by changing the method of calculating the intensity signal or the method of determining the feature points (e.g. by rule-based). . Therefore, the classification solution is modified in accordance with the modification by the classification unit 23, and the modified classification solution is stored in the DB.

The second problem is to calculate the intensity signal in some adaptive way and derive the feature points of this signal in order to determine the relevant length of the subsequence based on a predetermined adaptive algorithm. This is solved by

The third problem is solved by introducing the DB and mapping unit 24 into the classification system 20. These units enable classification system 20 to generate categories and category labels using appropriate numbers and textual information. In particular, the mapping unit 24 can generate mapping information by acquiring information about human movements from a database and/or the Internet, and the classification unit 23 can use the mapping information to improve classification accuracy. I can do it.

Categories can be automatically learned by the classification system 20, and new categories for new actions can be determined even when new action subsequences are performed, with less need for data.

As explained above, the classification system 20 can modify the predetermined algorithm based on the evaluation of the classification performed by the classification unit 23. Therefore, classification system 20 can more accurately classify subsequences.

Additionally, the preprocessor 21 can reduce information included in the raw video data and generate video data that includes information related to classification. Thereby, processing related to classification can be performed in less processing time, and classification accuracy can be improved.

Furthermore, the modification unit 25 includes an index indicating how well the classification means classifies elements that are already known to belong to the same category as part of the same category, and a default value for the defined problem. At least one of the following indicators may be used to evaluate the classification: a deviation from the number of categories, and an indicator of how well the system is accomplishing the overall task. Therefore, the classification system 20 can practically evaluate the classification. Here, the system includes a classification device.

Furthermore, the classification unit 23 can classify the subsequence (partial video data) as a type of human motion. As a result, classification system 20 can be used to detect human motion.

In particular, the generator 22 may calculate the intensity signal of the video data to determine a specific time domain. Here, the intensity signal indicates human motion. Since the characteristics of a person's motion can be defined as a simple intensity signal, the generation unit 22 can easily understand the characteristics of the person's motion.

Furthermore, the classification unit 23 can assign the classified subsequences (partial video data) to text labels. Therefore, the user of the classification system 20 can easily recognize the classification results.

(Embodiment 4)
Embodiment 4 of the present disclosure will be described below with reference to the drawings.

FIG. 13 shows an intention detection system 30. The intention detection system 30 includes a classification system 20, a human object analysis section 31, and an intention detection section 32. In summary, intent detection system 30 is a system coupled with an intent detection inference module. The processing of the units from the preprocessing section 21 to the modification section 25 is the same as that described in the third embodiment, so the explanation thereof will be omitted. An example of the recognition unit 15 in the second embodiment corresponds to the human object analysis unit 31 and the intention detection unit 32.

The human object analysis unit 31 analyzes the video data input by the preprocessing unit 21 and the subsequences generated by the generation unit 22, and detects various types of human parts in the subsequences. The detected human parts are, for example, the head, the right or left arm, the right or left leg, etc. Preferably, the human object analysis unit 31 can detect a portion used for a gesture indicating an instruction. The person object analysis section 31 outputs the detection result to the classification section 23. The classification unit 23 uses the detection results to classify the subsequences and improves classification accuracy.

The intention detection unit 32 receives the classification result from the classification unit 23, and uses this to detect the intention of the person in the video data. In this disclosure, "intention" can represent work on a certain object. The tasks include, for example, the task of grasping a certain object and the task of placing a certain object. When the intention detection system 30 is installed in a factory, the intention detection unit 32 detects the intention of the worker (for example, "expresses a feeling of wanting to grab a certain object", "expresses a feeling of wanting to be noticed"). , ``expressing the feeling of wanting to place the object,'' etc.) can be detected. Furthermore, "intent" can also represent instructions for machine operation. Operation of the machine can include, for example, moving, operating parts of the machine, or stopping these operations. The intention detection unit 32 outputs the result of intention detection. Examples of outputs include estimated subject activities and/or gestures for the subsequence being analyzed. Furthermore, the intention detection unit 32 may predict the next action and/or gesture of the person and output the prediction.

In this case, the intention detection unit 32 can detect the human intention using the classified subsequences (partial video data). Thereby, the intention detection system 30 can be applied to human activity support systems in various fields such as the industrial field and/or the medical field.

(Embodiment 5)
Embodiment 5 of the present disclosure will be described below with reference to the drawings. This embodiment describes a specific application of intent detection system 30.

FIG. 14 shows a machine including an intent detection system 30. Specifically, the machine 40 includes an intention detection system 30, a sensor S, a signal generator 41 and an optimizer controller 42. The processing of the intention detection system 30 is the same as that described in Embodiment 4, so the description thereof will be omitted. An example of machine 40 is a robot.

The sensor S acquires raw video data and inputs it to a preprocessor 21 within the intention detection system 30. For example, sensor S may be a video sensor.

The signal generator 41 receives the output of the intention detection section 32 in the intention detection system 30 and generates a control signal for controlling the operation of the machine 40, also taking into account the output of the intention detection section 32. For example, the signal generator 41 can determine the operation of the machine 40 according to the work determined by the intention detection unit 32, and can control the machine 40 according to the determined work. The signal generator 41 may receive other input signals from other sensors and/or parts of the machine, as shown in FIG. 14, and may also generate control signals in consideration of other input signals. Signal generator 41 functions as a controller for machine 40. For example, if the machine can move on the ground, the signal generator 41 can function as a trajectory planner and generate movement control signals as well as a planned trajectory. Additionally, signal generator 41 can receive signals from a part of machine 40 and generate reference signals to control that part. The signal generator 41 outputs the generated signal to the optimizer controller 42. Optimizer controller 42 receives control signals and processes the control signals as an optimizer. This is how machine 40 plans and controls its operation.

Figure 15 shows a particular application of machine 40 as a picking robot. The picking robot R includes an intention detection system 30 therein, and also includes a suction mechanism AM and a storage space. The suction mechanism AM sucks in items, and the sucked items are stored in a storage space that corresponds to the internal control of the picking robot R.

16A and 16B show an example of processing by the picking robot R instructed by a human gesture. 16A and 16B show a situation in which a worker W wants to issue instructions and commands to a picking robot R in a warehouse or factory. The picking robot R can monitor the worker W, acquire video data, and recognize the worker's gestures. Through the processes described in the third and fourth embodiments, the picking robot R classifies the worker's gestures and detects the worker's intention based on the classification. The picking robot R can perform the desired work using the intention detection result. The picking robot R may store the correspondence between the detected gestures (ie, instructions) of the worker W and the work performed by the picking robot R. Upon detecting the gesture, the picking robot R may perform a desired task based on the stored correspondence.

For example, in FIG. 16A, worker W is extending his right arm toward shelf S. FIG. 16A also shows that there are many different products on shelf S. Before the worker W's gesture, the picking robot R does not collect the products on the shelf S. However, when the worker W makes a gesture, the picking robot R classifies this gesture of the worker W and determines that this gesture corresponds to the process of sucking in the product on the shelf S. Then, the signal generator 41 of the picking robot R generates a control signal to move the picking robot R to a position near the shelf S, and causes the suction mechanism AM to suck up and collect the products on the shelf S.

As another example, in FIG. 16B, worker W is moving his left arm from the right side to the left side in FIG. 16B. The picking robot R classifies this gesture of the worker W and determines that this gesture corresponds to the process of stopping the work and leaving the shelf S. Then, the signal generator 41 in the picking robot R generates control signals for performing these operations.

In related technology, markers are often required for instructions to machines, even though attaching a marker to a person may be cumbersome. However, this disclosure discloses an advanced machine learning system that is applicable to a variety of machines and can provide a "markerless solution." Therefore, the burden of attaching the marker to a person can be avoided.

Further, the signal generator 41 (controller) controls the operation of the machine 40 based on the human intention detected by the intention detection section 32. Therefore, the machine 40 can support the work of the worker.

Note that the present invention is not limited to the above-described embodiments, and may be modified as appropriate without departing from the spirit of the present invention. For example, instead of the modification unit 25, another unit within the classification system 20 or a device external to the classification system 20 may evaluate the classification performed by the classification unit 23.

Since different human motions may be performed with their motions overlapping in time, the generated partial video data (or subsequences) may overlap with each other in time in embodiments 1 and 2.

FIG. 8A shows examples of the person P's "raise left arm," "pass the object," and "relax." However, it goes without saying that examples of human actions are not limited to these. For example, the human motion to be detected may be "raising the left arm near an object", "raising the right arm", "pointing with the index finger", "making a special gesture with the hand", etc.

The present disclosure provides that the main information of a data frame is summarized in a small number of somehow related points in a two- or three-dimensional space that change positions in space, and that the images of these points are summarized at a certain time. can be applied to the uses given in the steps.

The present disclosure relates to classification systems, methods, and programs for various purposes that can classify motion patterns obtained from point data computed from regularly or irregularly sampled sequences of movie frames. This technical system helps to determine the movement patterns of the person performing the act and to classify the movement patterns accordingly. This may be applied in intention detection systems where correctly classified and labeled motion subsequences play an important role for further processing, e.g. planning assistance to a human. Specifically, it can be used in various situations such as factories, shopping malls, warehouses, cafeteria kitchens, or construction sites. Additionally, it can be used to analyze human motion in sports-related or other activities. It can also be applied to characterize very general dynamic patterns. However, the application of the present disclosure is not necessarily limited to this field.

Next, an example of the hardware configuration of the apparatus described in the above embodiments will be described below with reference to FIG. 17.

FIG. 17 is a block diagram showing a configuration example of an information processing device. The information processing device 90 includes a network interface 91, a processor 92, and a memory 93, as shown in FIG. The network interface 91 can send and receive data to and from other devices via wireless communication.

The processor 92 executes the processing performed by the information processing device 90 described with reference to the sequence diagrams and flowcharts of the above embodiments by loading and executing software (computer programs) from the memory 93. The processor 92 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). Processor 92 may include multiple processors.

The memory 93 is configured by a combination of volatile memory and nonvolatile memory. Memory 93 may include storage spaced apart from processor 92. In this case, processor 92 may access memory 93 via an I/O interface (not shown).

In the example shown in FIG. 17, memory 93 is used to store software modules. The processor 92 can perform the processing performed by the information processing apparatus described in the above embodiments by reading a software module group from the memory 93 and executing the read software module.

As described above with reference to FIG. 17, each processor included in the information processing device of the above embodiment executes one or more programs including a group of instructions, and executes the program as described above with reference to the drawings. have the computer execute the algorithm.

Furthermore, the information processing device 90 may include a network interface. The network interface is used for communication with other network node devices that make up the communication system. The network interface may include, for example, a network interface card (NIC) that is compliant with the IEEE 802.3 series. The information processing device 90 may receive input feature maps or transmit output feature maps using a network interface.

In the above example, the program may be stored and provided to the computer using any type of non-transitory computer-readable medium. Non-transitory computer-readable media includes any type of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., floppy disks, magnetic tape, hard disk drives, etc.), magneto-optical storage media (e.g., magneto-optical disks), and compact disc read only memory (CD-ROM) media. ), CD-R (compact disc recordable), CD-R/W (compact disc rewritable), semiconductor memory (e.g. mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory) )and so on. The program may be provided to a computer using any type of temporary computer-readable medium. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable medium can provide the program to the computer over a wired (eg, wire, fiber optic) or wireless communication link.

Although some or all of the above embodiments can be described as in the following additional notes, the present disclosure is not limited thereto.
(Additional note 1)
generating means for determining a specific time region of video data based on a predetermined algorithm and generating partial video data from which the video data is extracted in the specific time region;
Classifying means for classifying the partial video data generated by the generating means;
modification means for modifying the predetermined algorithm based on the evaluation of the classification performed by the classification means;
A classification device comprising:
(Additional note 2)
further comprising pre-processing means for reducing information contained in the raw data and generating the video data including information related to the classification;
The classification device described in Appendix 1.
(Additional note 3)
The modification means includes an index indicating how well the classification means classifies elements that are already known to belong to the same category as part of the same category, and a default index for the defined problem. evaluating the classification using at least one indicator of a deviation from the number of categories and an indicator of how well a system including the classification device is accomplishing the overall task;
The classification device according to appendix 1 or 2.
(Additional note 4)
The classification means classifies the partial video data as a type of human motion.
The classification device according to any one of Supplementary Notes 1 to 3.
(Appendix 5)
the generating means calculates an intensity signal of the video data to determine the specific time domain, the intensity signal indicating human motion;
The classification device described in Appendix 4.
(Appendix 6)
The classification means assigns the classified partial video data to a text label.
The classification device according to appendix 4 or 5.
(Appendix 7)
further comprising an intention detection means for detecting a human intention using the classified partial video data;
The classification device according to any one of Supplementary Notes 4 to 6.
(Appendix 8)
further comprising a controller that controls the operation of the machine based on the human intention detected by the intention detection means;
The classification device described in Appendix 7.
(Appendix 9)
recognition means for recognizing video data including a task and thereby determining said task;
a controller that determines the operation of a machine according to the determined work and controls the machine according to the determined work;
A control device comprising:
(Appendix 10)
further comprising a classification means for classifying the video data and inputting the classified video data to the recognition means;
The control device according to appendix 9.
(Appendix 11)
generating means for determining a specific time region of video data based on a predetermined algorithm and generating partial video data from which the video data is extracted in the specific time region;
further comprising a modifying means for modifying the predetermined algorithm based on the evaluation of the classification performed by the classifying means,
the partial video data is recognized by the recognition means;
The control device according to appendix 10.
(Appendix 12)
The modification means includes an index indicating how well the classification means classifies elements that are already known to belong to the same category as part of the same category, and a default index for the defined problem. evaluating the classification using at least one indicator of a deviation from the number of categories and an indicator of how well a system including the classification device is accomplishing the overall task;
The control device according to appendix 11.
(Appendix 13)
The classification means classifies the video data as a type of human motion.
The control device according to any one of Supplementary Notes 10 to 12.
(Appendix 14)
determining a specific time region of video data based on a predetermined algorithm, and generating partial video data in which the video data is extracted in the specific time region;
classifying the partial video data;
modifying the predetermined algorithm based on the classification evaluation;
Classification methods including.
(Appendix 15)
recognizing video data including a task and thereby determining said task;
determining an operation of a machine according to the determined work, and controlling the machine according to the determined work;
control methods including.
(Appendix 16)
determining a specific time region of video data based on a predetermined algorithm, and generating partial video data in which the video data is extracted in the specific time region;
classifying the partial video data;
modifying the predetermined algorithm based on the classification evaluation;
A non-transitory computer-readable medium that stores a program that causes a computer to execute.
(Appendix 17)
recognizing video data including a task and thereby determining said task;
determining an operation of a machine according to the determined work, and controlling the machine according to the determined work;
A non-transitory computer-readable medium that stores a program that causes a computer to execute.

It will be apparent to those skilled in the art that many variations and/or modifications may be made to this disclosure, as illustrated in particular embodiments, without departing from the spirit or scope of this disclosure as broadly described. will be understood. Accordingly, this embodiment is to be considered in all respects as illustrative and not restrictive.

10 Classification device 11 Generation section 12 Classification section 13 Modification section 14 Control device 15 Recognition section 16 Controller 20 Classification system 21 Preprocessing section 22 Generation section 23 Classification section 24 Mapping section 25 Modification section 26 Calculation section 27 Signal analysis section 28 Determination section 29 Subsequence generation section 30 Intention detection system 31 Person object analysis section 32 Intention detection section 40 Machine 41 Signal generator 42 Optimizer controller

FIG. 8B shows examples of categories and category labels corresponding to subsequences (1) to (3). The category of subsequence (1) is "mp31," the category of subsequence (2) is "mp76," and the category of subsequence ( 3 ) is "mp21." The classification unit 23 uses the DB to set these category numbers. Further, the category label of subsequence (1) is "raise left arm," the category of subsequence (2) is "pass the object," and the category of subsequence ( 3 ) is "relax." The classification unit 23 uses the text information generated by the mapping unit 24 to set these category labels. In this manner, classification system 20 defines labels for subsequences.

Claims (17)

generating means for determining a specific time region of video data based on a predetermined algorithm and generating partial video data from which the video data is extracted in the specific time region;
Classifying means for classifying the partial video data generated by the generating means;
modification means for modifying the predetermined algorithm based on the evaluation of the classification performed by the classification means;
A classification device comprising: further comprising pre-processing means for reducing information contained in the raw data and generating the video data including information related to the classification;
The classification device according to claim 1. The modification means includes an index indicating how well the classification means classifies elements that are already known to belong to the same category as part of the same category, and a default index for the defined problem. evaluating the classification using at least one indicator of a deviation from the number of categories and an indicator of how well a system including the classification device is accomplishing the overall task;
The classification device according to claim 1 or 2. The classification means classifies the partial video data as a type of human motion.
A classification device according to any one of claims 1 to 3. the generating means calculates an intensity signal of the video data to determine the specific time domain, the intensity signal indicating human motion;
The classification device according to claim 4. The classification means assigns the classified partial video data to a text label.
The classification device according to claim 4 or 5. further comprising an intention detection means for detecting a human intention using the classified partial video data;
The classification device according to any one of claims 4 to 6. further comprising a controller that controls the operation of the machine based on the human intention detected by the intention detection means;
The classification device according to claim 7. recognition means for recognizing video data including a task and thereby determining said task;
a controller that determines the operation of a machine according to the determined work and controls the machine according to the determined work;
A control device comprising: further comprising a classification means for classifying the video data and inputting the classified video data to the recognition means;
The control device according to claim 9. generating means for determining a specific time region of video data based on a predetermined algorithm and generating partial video data from which the video data is extracted in the specific time region;
further comprising a modifying means for modifying the predetermined algorithm based on the evaluation of the classification performed by the classifying means,
the partial video data is recognized by the recognition means;
The control device according to claim 10. The modification means includes an index indicating how well the classification means classifies elements that are already known to belong to the same category as part of the same category, and a default index for the defined problem. evaluating the classification using at least one indicator of a deviation from a number of categories and an indicator of how well a system including the control device is accomplishing an overall task;
The control device according to claim 11. The classification means classifies the video data as a type of human motion.
The control device according to any one of claims 10 to 12. determining a specific time region of video data based on a predetermined algorithm, and generating partial video data in which the video data is extracted in the specific time region;
classifying the partial video data;
modifying the predetermined algorithm based on the classification evaluation;
Classification methods including. recognizing video data including a task and thereby determining said task;
determining an operation of a machine according to the determined work, and controlling the machine according to the determined work;
control methods including. determining a specific time region of video data based on a predetermined algorithm, and generating partial video data in which the video data is extracted in the specific time region;
classifying the partial video data;
modifying the predetermined algorithm based on the classification evaluation;
A non-transitory computer-readable medium that stores a program that causes a computer to execute. recognizing video data including a task and thereby determining said task;
determining an operation of a machine according to the determined work, and controlling the machine according to the determined work;
A non-transitory computer-readable medium that stores a program that causes a computer to execute.

Images