WO2023042592A1

WO2023042592A1 - Method and apparatus for determining abnormal behaviour during cycle

Info

Publication number: WO2023042592A1
Application number: PCT/JP2022/030915
Authority: WO
Inventors: Masaharu Morimoto; Isaac PEK; Derek Ng; Hayato CHISHAKI
Original assignee: Nec Corporation
Priority date: 2021-09-14
Filing date: 2022-08-15
Publication date: 2023-03-23
Also published as: JP2024534931A; CN117769713A

Abstract

The present disclosure provides a method and an apparatus for determining abnormal behaviour during a cycle, the method comprising: identifying a set of movements during a first cycle which defines a first cycle of movements; identifying a set of movements during a second cycle which defines a second cycle of movements; and comparing the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle.

Description

METHOD AND APPARATUS FOR DETERMINING ABNORMAL BEHAVIOUR DURING CYCLE

　　The present invention relates broadly, but not exclusively, to a method and an apparatus for determining abnormal behaviour during a cycle.

　　For manufacturers, cycle time is an important indicator to measure the productivity in their assembly lines.

　　One cycle at each workbench typically consists of a series of actions such as, mounting components on the board, tightening a screw or putting the cover for packaging, etc.

　　Traditionally, cycle time is manually measured by line managers using a stop-watch. Since the measurement is done by sampling in such cases, it is difficult to get statistics based on long-term and continuous monitoring results, let alone determine abnormal behaviour during a cycle.

　　Video analytics can help to estimate the cycle time instead of solely relying on manual effort. Behaviour analytics especially, has the potential to detect the series of actions related to the work process in assembly lines.

　　This disclosure is related to abnormal behaviour determination methods using production time estimation.

　　Herein disclosed are embodiments of a devices and methods for determining abnormal behaviour during a cycle that addresses one or more of the above problems.

　　Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

　　According to a first aspect, there is a method for determining abnormal behaviour during a cycle, comprising: identifying a set of movements during a first cycle which defines a first cycle of movements; identifying a set of movements during a second cycle which defines a second cycle of movements; and comparing the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle.

　　According to a second aspect, there is an apparatus for determining abnormal behaviour during a cycle, the apparatus comprising: at least one processor; and at least one memory including computer program code; wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: identify a set of movements during a first cycle which defines a first cycle of movements; identify a set of movements during a second cycle which defines a second cycle of movements; and compare the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle.

　　According to the present disclosure, it is possible to provide devices and methods for determining abnormal behaviour during a cycle that addresses one or more of the above problems.

　　The accompanying Figures, where like reference numerals refer to identical or functionally similar elements throughout the separate views and which together with the detailed description below are incorporated in and form part of the specification, serve to illustrate various embodiments and to explain various principles and advantages in accordance with a present embodiment, by way of non-limiting example only.

　　Embodiments of the invention will be better understood and readily apparent to one of ordinary skill in the art from the following written description, by way of example only, and in conjunction with the drawings, in which:

Fig. 1 shows a system for determining abnormal behaviour during a cycle according to an aspect of the present disclosure.

Fig. 2A shows two time series during which two cycles of movements can be identified according to an embodiment of the present disclosure.

Fig. 2B shows a flow chart of how abnormal behaviour detection may be carried out according to an embodiment of the present disclosure.

Fig. 3 shows how a first cycle of movements and a second cycle of movements go through Canonical Time Wrapping (CTW).

Fig. 4 shows how a first cycle of movements and a second cycle of movements go through Longest Common Subsequence (LCSS).

Fig. 5 shows how a flow chart of how abnormal behaviour detection may be carried out according to an embodiment of the present disclosure.

Fig. 6 shows how long sideslip can be detected according to an embodiment of the present disclosure.

Fig. 7 shows how long sideslip can be detected according to an embodiment of the present disclosure.

Fig. 8 shows how long sideslip can be detected according to an embodiment of the present disclosure.

Fig. 9 shows how long sideslip can be detected according to another embodiment of the present disclosure.

Fig. 10 shows how long sideslip can be detected according to one embodiment of the present disclosure.

Fig. 11 shows an exemplary computing device that may be used to execute the method of the earlier figures.

(Terms Description)
　　Subject- a subject may be any suitable type of entity, which may include a person, a worker and a user.

　　The term target or target subject is used herein to identify a person, a user or worker that is of interest. The target subject may be one that is selected by a user input or one who is identified to be of interest.

　　A subject or an identified subject is used herein to relate to a person who is related to the target subject (e.g. partner or someone with similar skillset). For example, in the context of determining abnormal behaviour during a cycle, the subject is someone who may be considered to have the similar skillset or experience as the target.

　　A user who is registered to an abnormal behaviour determining server will be called a registered user. A user who is not registered to the abnormal behaviour determining server will be called a non-registered user. It is possible for the user to obtain abnormal behaviour determination of any subject or any cycle. Abnormal behaviour determining server - The abnormal behaviour determining server is a server that hosts software application programs for receiving inputs, processing data and objectively providing graphical representation. The abnormal behaviour determining server communicates with any other servers (e.g., a remote assistance server) to manage requests. The abnormal behaviour determining server communicates with a remote assistance server to receive ground rules or predetermined movements. Abnormal behaviour determining servers may use a variety of different protocols and procedures in order to manage the data and provide a graphical representation. In various embodiments below, abnormal behaviour may refer to actions performed by a worker (or subject) to assemble one motherboard, for example, are different from usual actions. For example, the subject may usually take 30 seconds to assemble one motherboard but in the abnormal cycle he takes 90 seconds. The abnormal behaviour would pick up this abnormal cycle.

　　By picking up this abnormal behaviour, one may understand that it took the subject 60 seconds more in tightening a screw because a distraction or missing steps in a cycle.

　　Advantageously, this allows one to detect if assembly of parts is done correctly, especially those including low quality components.

　　The abnormal behaviour determining server is usually managed by a provider that may be an entity (e.g. a company or organization) which operates to process requests, manage data and receive/ display graphical representations that are useful to a situation. The server may include one or more computing devices that are used for processing graphical representation requests and providing customisable services depending on situations.

　　An abnormal behaviour determining account - an abnormal behaviour determining account is an account of a user who is registered at an abnormal behaviour determining server. In certain circumstances, the abnormal behaviour determining account is not required to use the remote assistance server. An abnormal behaviour determining account includes details (e.g., name, address, vehicle etc.) of a user. A cycle time is a period of time for each first movement and second movement.

　　The abnormal behaviour determining server manages abnormal behaviour determining accounts of users and the interactions between users and other external servers, along with the data that is exchanged.

(Detailed Description)
　　Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

　　It is to be noted that the discussions contained in the "Background" section and that above relating to prior art arrangements relate to discussions of devices which form public knowledge through their use. Such should not be interpreted as a representation by the present inventor(s) or the patent applicant that such devices in any way form part of the common general knowledge in the art.

　　The system 100

　　Fig. 1 illustrates a block diagram of a system 100 for abnormal behaviour determining. The system 100 comprises a requestor device 102, an abnormal behaviour determining server 108, a remote assistance server 140, remote assistance hosts 150A to 150N, sensors 142A to 142N, and Database 109.

　　The requestor device 102 is in communication with an abnormal behaviour determining server 108 and/or a remote assistance server 140 via a

connection

116 and 121, respectively. The

connection

116 and 121 may be wireless (e.g., via NFC communication, Bluetooth, etc.) or over a network (e.g., the Internet). The

connection

116 and 121 may also be that of a network (e.g., the Internet).

　　The abnormal behaviour determining server 108 is further in communication with the remote assistance server 140 via a connection 120. The connection 120 may be over a network (e.g., a local area network, a wide area network, the Internet, etc.). In one arrangement, the abnormal behaviour determining server 108 and the remote assistance server 140 are combined and the connection 120 may be an interconnected bus.

　　The remote assistance server 140, in turn, is in communication with the remote assistance hosts 150A to 150N via respective connections 122A to 122N. The connections 122A to 122N may be a network (e.g., the Internet).

　　The remote assistance hosts 150A to 150N are servers. The term host is used herein to differentiate between the remote assistance hosts 150A to 150N and the remote assistance server 140. The remote assistance hosts 150A to 150N are collectively referred to herein as the remote assistance hosts 150, while the remote assistance host 150 refers to one of the remote assistance hosts 150. The remote assistance hosts 150 may be combined with the remote assistance server 140.

　　In an example, the remote assistance host 150 may be one managed by a factory and the remote assistance server 140 is a central server that manages productivity at an organization level and decides which of the remote assistance hosts 150 to forward data or retrieve data like image inputs.

　　Sensors 142A to 142N are connected to the remote assistance server 140 or the abnormal behaviour determining server 108 via respective connections 144A to 144N or 146A to 146N. The sensors 142A to 142N are collectively referred to herein as the sensors 142. The connections 144A to 144N are collectively referred to herein as the connections 144, while the connection 144 refers to one of the connections 144. Similarly, the connections 146A to 146N are collectively referred to herein as the connections 146, while the connection 146 refers to one of the connections 146. The connections 144 and 146 may be wireless (e.g., via NFC communication, Bluetooth, etc.) or over a network (e.g., the Internet). The sensors 142may be one of an image capturing device, video capturing device, and motion sensor and may be configured to send an input depending its type, to at least one of the abnormal behaviour determining server 108. The sensors 142 captures motions of a subject (or a worker) performing a sequence of actions and sends the captured motions to the server 108 which will determine if there is an abnormal behaviour.

　　In the illustrative embodiment, each of the devices 102 and 142; and the

servers

108, 140, and 150 provides an interface to enable communication with other connected devices 102 and 142 and/or

servers

108, 140, and 150. Such communication is facilitated by an application programming interface ("API"). Such APIs may be part of a user interface that may include graphical user interfaces (GUIs), Web-based interfaces, programmatic interfaces such as application programming interfaces (APIs) and/or sets of remote procedure calls (RPCs) corresponding to interface elements, messaging interfaces in which the interface elements correspond to messages of a communication protocol, and/or suitable combinations thereof.

　　Use of the term 'server' herein can mean a single computing device or a plurality of interconnected computing devices which operate together to perform a particular function. That is, the server may be contained within a single hardware unit or be distributed among several or many different hardware units.

　　The remote assistance server 140

　　The remote assistance server 140 is associated with an entity (e.g. a factory or a company or organization or moderator of the service). In one arrangement, the remote assistance server 140 is owned and operated by the entity operating the server 108. In such an arrangement, the remote assistance server 140 may be implemented as a part (e.g., a computer program module, a computing device, etc.) of server 108.

　　The requestor device 102

　　The requestor device 102 is associated with a subject (or requestor) who is a party to a request that starts at the requestor device 102. The requestor may be a concerned member of the public who is assisting to get data necessary to obtain a graphical representation of a network graph. The requestor device 102 may be a computing device such as a desktop computer, an interactive voice response (IVR) system, a smartphone, a laptop computer, a personal digital assistant computer (PDA), a mobile computer, a tablet computer, and the like.

　　In one example arrangement, the requestor device 102 is a computing device in a watch or similar wearable and is fitted with a wireless communications interface.

　　The abnormal behaviour determining server 108

　　The abnormal behaviour determining server 108 is as described above in the terms description section.

　　The abnormal behaviour determining server 108 is configured to process processes relating to compare the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle.

　　The remote access hosts 150

　　The remote access host 150 is a server associated with an entity (e.g. a company or organization) which manages (e.g. establishes, administers) information regarding information relating to a subject or a member of an organisation. In one embodiment, the server stores information relating to facial recognition of the subjects (or workers).

　　In one arrangement, the entity is an organisation. Therefore, each entity operates a remote access host 150 to manage the resources by that entity. In one arrangement, a remote access host 150 receives an alert signal that a target subject is in motion. The remote access host 150 may then arrange to send resources to the location identified by the location information included in the alert signal. For example, the host may be one that is configured to obtain relevant video or image input for processing.

　　Advantageously, such information is valuable to detect is abnormal behaviour during a cycle at factory assembly lines. This disclosure uses correlations between hand positions and start/end timings of cycles. As such, a more accurate detection of abnormal behaviour can be obtained.

　　Hand positions are better suited to identify start/end timings of cycles in factory situation because objects move from left to right or vice versa on belt conveyers at assembly lines. Thus, the actual position can generate better features for those situations. In various embodiments, it can be appreciated that the disclosure can be used for cellular production methods.

　　In various embodiments, a cycle can be detected for example, by detecting start and end of actions of workers. One example is position-based action detection if the start and end actions are related to specific positions as detected by the sensors 142. Another example is to use action recognition technique capturing movements of hands over time and recording such actions as start, end or other actions.

　　Sensor 142

　　The sensor 142 is associated with a user associated with the requestor device 102. More details of how the sensor may be utilised will be provided below.

　　Fig. 2A shows two time series during which two cycles of movements can be identified according to an embodiment of the present disclosure. 202 shows how a set of movements during a first cycle which defines a first cycle of movements. 204 shows how a set of movements during a second cycle which defines a second cycle of movements.

　　Fig. 2B shows a flow chart of how abnormal behaviour detection may be carried out according to an embodiment of the present disclosure. At step 222, time series hand movement may be identified by a time-series hand position generator. Step 224 may follow step 222. At step 224, a cycle defining each cycle of movement may be estimated by a cycle time generator. Step 226 may follow step 224. At step 226, the first cycle of movements is comparied with the second cycle of movement to determine if there is abnormal behaviour during the second cycle. Step 228 may follow step 226. At step 228, if it is determined that there is abnormal behaviour during the second cycle, it is reflected to the cycle time viewer which can be any image viewer.

　　Fig. 3 shows how a first cycle of movements, as represented as 502, and a second cycle of movements, as represented as 504, go through Canonical Time Wrapping (CTW). The resultant outcome 505 should be square and diagonal if the first cycle and the second cycle is similar with normal behaviors.

　　In an embodiment, there may be abnormal behavior between the first cycle of movements 502 and the second cycle of movements 506, the resultant outcome 508 after performing CTW may include long sideslips (or flat portions) that are indicative that there are abnormal behaviour between the two cycles of movements. The long sideslips 512a may be represented as 510a in a different colour to a user device.

　　Fig. 4 shows how a first cycle of movements, as represented as 602, and a second cycle of movements, as represented as 604, go through Longest Common Subsequence (LCW). The resultant outcome 605 should be square and diagonal if the first cycle and the second cycle is similar with normal behaviors.

　　In an embodiment, there may be abnormal behavior between the first cycle of movements 602 and the second cycle of movements 606, the resultant outcome 608 after performing LCSS may include long sideslips (or flat portions) that are indicative that there are abnormal behaviour between the two cycles of movements. The long sideslips 612a may be represented as 610a in a different colour to a user device.

　　Fig. 5 shows how a flow chart of how abnormal behaviour detection may be carried out according to an embodiment of the present disclosure. At step 702, the flow chart starts. Step 704 may follow step 702. At step 704, warping paths of the cycle of movements may be generated. Step 706 may follow step 704. At step 706, it is determine if there are any long sideslips to determine if there is abnormal behaviour during the second cycle. Step 708 may follow step 706. At step 708, when it is determined that there are long sideslips, it is reflected to the cycle time viewer which can be any image viewer as anomalies.

　　Fig. 6 shows how long sideslip can be detected according to an embodiment of the present disclosure. 802 may represent a normal warping path as a reference. 810 represents a warping path that is obtained according to the flow chart shown in Fig. 5. An alert or anomaly is sent if there are more than N-consecutive same values of vertical coordinates (*, y) or horizontal coordinates (x, *) in the warping path 810. This represents a long sideslip 806.

　　Fig. 7 shows how long sideslip can be detected according to an embodiment of the present disclosure. 902 may represent a normal warping path as a reference. 910 represents a warping path that is obtained according to the flow chart shown in Fig. 5. A threshold 904 may be one that is obtained by height (H) over thickness (T) of the threshold. 906 represents a warping path that is obtained according to the flow chart shown in Fig. 5. For 910, it may represent a warping path that is obtained according to the flow chart shown in Fig. 5. Portions of 910 may be obtained to see if the slope of that portion is less than the threshold (=H/T). If it is determined that the slope of that portion is less than the threshold (=H/T), an alert or anomaly is sent. This represents a long sideslip 916.

　　Fig. 8 shows how long sideslip can be detected according to an embodiment of the present disclosure. 1002 may represent a normal warping path as a reference. 1010 represents a warping path that is obtained according to the flow chart shown in Fig. 5. According to the embodiment in Fig. 8, an alert is given out if warping path 1002 goes to out of the band.1002 may be represented by y = ax. 1004 may be represented by y = ax + b and 1006 may be represented by y = ax - b.

　　In Fig. 9, portions of the warping path may be obtained to see if the slope of that portion is out of band. If it is determined that the slope of that portion is out of band, an alert or anomaly is sent. This represents a long sideslip 1008.

　　In Fig. 10, portions of the warping path may be obtained to see if the end of the warping path is in the path. If it is determined that the portions are in the band, stop.

　　Fig. 11 depicts an exemplary computing device 1300, hereinafter interchangeably referred to as a computer system 1300, where one or more such computing devices 1300 may be used to execute the methods shown above. The exemplary computing device 1300 can be used to implement the system 100 shown in Fig. 1. The following description of the computing device 1300 is provided by way of example only and is not intended to be limiting.

　　As shown in Fig. 11, the example computing device 1300 includes a processor 1307 for executing software routines. Although a single processor is shown for the sake of clarity, the computing device 1300 may also include a multi-processor system. The processor 1307 is connected to a communication infrastructure 1306 for communication with other components of the computing device 1300. The communication infrastructure 1306 may include, for example, a communications bus, cross-bar, or network.

　　The computing device 1300 further includes a main memory 1308, such as a random access memory (RAM), and a secondary memory 1310. The secondary memory 1310 may include, for example, a storage drive 1312, which may be a hard disk drive, a solid state drive or a hybrid drive and/or a removable storage drive 1317, which may include a magnetic tape drive, an optical disk drive, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), or the like. The removable storage drive 1317 reads from and/or writes to a removable storage medium 1377 in a well-known manner. The removable storage medium 1377 may include magnetic tape, optical disk, non-volatile memory storage medium, or the like, which is read by and written to by removable storage drive 1317. As will be appreciated by persons skilled in the relevant art(s), the removable storage medium 1377 includes a computer readable storage medium having stored therein computer executable program code instructions and/or data.

　　In an alternative implementation, the secondary memory 1310 may additionally or alternatively include other similar means for allowing computer programs or other instructions to be loaded into the computing device 1300. Such means can include, for example, a removable storage unit 1322 and an interface 1350. Examples of a removable storage unit 1322 and interface 1350 include a program cartridge and cartridge interface (such as that found in video game console devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a removable solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), and other removable storage units 1322 and interfaces 1350 which allow software and data to be transferred from the removable storage unit 1322 to the computer system 1300.

　　The computing device 1300 also includes at least one communication interface 1327. The communication interface 1327 allows software and data to be transferred between computing device 1300 and external devices via a communication path 1326. In various embodiments of the inventions, the communication interface 1327 permits data to be transferred between the computing device 1300 and a data communication network, such as a public data or private data communication network. The communication interface 1327 may be used to exchange data between different computing devices 1300 which such computing devices 1300 form part an interconnected computer network. Examples of a communication interface 1327 can include a modem, a network interface (such as an Ethernet card), a communication port (such as a serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB), an antenna with associated circuitry and the like. The communication interface 1327 may be wired or may be wireless. Software and data transferred via the communication interface 1327 are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communication interface 1327. These signals are provided to the communication interface via the communication path 1326.

　　As shown in Fig. 11, the computing device 1300 further includes a display interface 1302 which performs operations for rendering images to an associated display 1301 and an audio interface 1352 for performing operations for playing audio content via associated speaker(s) 1357.

　　As used herein, the term "computer program product" may refer, in part, to removable storage medium 1377, removable storage unit 1322, a hard disk installed in storage drive 1312, or a carrier wave carrying software over communication path 1326 (wireless link or cable) to communication interface 1327. Computer readable storage media refers to any non-transitory, non-volatile tangible storage medium that provides recorded instructions and/or data to the computing device 1300 for execution and/or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-rayTM Disc, a hard disk drive, a ROM or integrated circuit, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), a hybrid drive, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computing device 1300. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and/or data to the computing device 1300 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

　　The computer programs (also called computer program code) are stored in main memory 1308 and/or secondary memory 1310. Computer programs can also be received via the communication interface 1327. Such computer programs, when executed, enable the computing device 1300 to perform one or more features of embodiments discussed herein. In various embodiments, the computer programs, when executed, enable the processor 1307 to perform features of the above-described embodiments. Accordingly, such computer programs represent controllers of the computer system 1300.

　　Software may be stored in a computer program product and loaded into the computing device 1300 using the removable storage drive 1317, the storage drive 1312, or the interface 1350. The computer program product may be a non-transitory computer readable medium. Alternatively, the computer program product may be downloaded to the computer system 1300 over the communications path 1327. The software, when executed by the processor 1307, causes the computing device 1300 to perform the necessary operations to execute the method as described above.

　　It is to be understood that the embodiment of Fig. 11 is presented merely by way of example to explain the operation and structure of the system 100. Therefore, in some embodiments one or more features of the computing device 1300 may be omitted. Also, in some embodiments, one or more features of the computing device 1300 may be combined together. Additionally, in some embodiments, one or more features of the computing device 1300 may be split into one or more component parts.

　　It will be appreciated by a person skilled in the art that numerous variations and/or modifications may be made to the present invention as shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.

　　Further, the whole or part of the embodiments disclosed above can be described as, but not limited to, the following supplementary notes.
(Supplementary note 1)
　　A method for determining abnormal behaviour during a cycle, comprising:
　　identifying a set of movements during a first cycle which defines a first cycle of movements;
　　identifying a set of movements during a second cycle which defines a second cycle of movements; and
　　comparing the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle.
(Supplementary note 2)
　　The method according to Supplementary note 1, further comprising:
　　performing time wrapping on the first cycle of movements and the second cycle of movements.
(Supplementary note 3)
　　The method according to Supplementary note 2, wherein the step of performing time wrapping includes Dynamic Time Warping (DTW) to find a warping path that describes best matching between the first cycle and the second cycle.
(Supplementary note 4)
　　The method according to Supplementary note 2, wherein the step of performing time wrapping includes Canonical Time Warping (CTW) to align the first cycle and the second cycle under rigid registration of the feature space.
(Supplementary note 5)
　　The method according to Supplementary note 2, wherein the step of performing time wrapping includes Longest Common Subsequence to find similar parts of two sequences.
(Supplementary note 6)
　　The method according to any one of Supplementary notes 2 to 5, wherein the step of performing time wrapping includes calculating a distance matrix and a warping path.
(Supplementary note 7)
　　The method according to Supplementary note 6, further comprising determining whether or not there is a sideslip between the first cycle and the second cycle in response to the calculation of the warping path.
(Supplementary note 8)
　　The method according to Supplementary note 7, wherein the step of comparing the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle includes determining there is abnormal behaviour during the second cycle when it is determined that there is the sideslip between the first cycle and the second cycle.
(Supplementary note 9)
　　The method according to any one of the preceding Supplementary notes, when it is determined that there is abnormal behaviour during the second cycle, the method further comprising representing the abnormal behavior to a user device.
(Supplementary note 10)
　　An apparatus for determining abnormal behaviour during a cycle, the apparatus comprising:
　　at least one processor; and
　　at least one memory including computer program code; wherein
　　the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to:
　　identify a set of movements during a first cycle which defines a first cycle of movements;
　　identify a set of movements during a second cycle which defines a second cycle of movements; and
　　compare the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle.
(Supplementary note 11)
　　The apparatus according to Supplementary note 10, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to perform time wrapping on the first cycle of movements and the second cycle of movements.
(Supplementary note 12)
　　The apparatus according to Supplementary note 11, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to perform Dynamic Time Warping (DTW) to find a warping path that describes best matching between the first cycle and the second cycle.
(Supplementary note 13)
　　The apparatus according to Supplementary note 11, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to perform Canonical Time Warping (CTW) to align the first cycle and the second cycle under rigid registration of the feature space.
(Supplementary note 14)
　　The apparatus according to Supplementary note 11, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to perform Longest Common Subsequence to find similar parts of two sequences.
(Supplementary note 15)
　　The apparatus according to any one of Supplementary notes 11 to 14, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to calculate a distance matrix and a warping path.
(Supplementary note 16)
　　The apparatus according to Supplementary note 15, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to determine whether or not there is a sideslip between the first cycle and the second cycle in response to the calculation of the warping path.
(Supplementary note 17)
　　The apparatus according to Supplementary note 16, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to determine if there is abnormal behaviour during the second cycle includes determining there is abnormal behaviour during the second cycle when it is determined that there is the sideslip between the first cycle and the second cycle.
(Supplementary note 18)
　　The apparatus according to any one of Supplementary notes 11 to 16, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to represent the abnormal behavior to a user device.

　　This application is based upon and claims the benefit of priority from Singapore Patent Application No. 10202110130P, filed on September 14, 2021, the disclosure of which is incorporated herein in its entirety by reference.

100　　system
102　　requestor device
108　　abnormal behaviour determining server
109　　database
116　　connection
120　　connection
121　　connection
122A to 122N　　connection
140　　 remote assistance server
142A to 142N　　Sensor
144A to 144N　　connection
146A to 146N　　connection
150A to 150N　　remote access host
202　　how a set of movements during a first cycle
204　　how a set of movements during a second cycle
502　　first cycle of movements
504　　second cycle of movements
505　　resultant outcome
506　　second cycle of movements
508　　resultant outcome
512a　　long sideslip
510a　　long sideslip
602　　first cycle of movements
604　　second cycle of movements
605　　resultant outcome
606　　second cycle of movements
608　　resultant outcome
610a　　long sideslip
612a　　long sideslip
802　　normal warping path
806　　long sideslip
810　　warping path
902　　normal warping path
904　　threshold
906　　warping path
910　　warping path
916　　long sideslip
1002　　normal warping path
1004　　warping path
1006　　warping path
1008　　long sideslip
1202　　portions of the warping path
1204　　portions of the warping path
1302　　display interface
1306　　communication infrastructure
1307　　processor
1308　　main memory
1310　　secondary memory
1312　　storage drive
1317　　removable storage drive
1322　　removable storage unit
1326　　communication path
1327　　communication interface
1301　　display
1350　　interface
1352　　audio interface
1357　　speaker(s)
1377　　removable storage medium

Claims

　　A method for determining abnormal behaviour during a cycle, comprising:
　　identifying a set of movements during a first cycle which defines a first cycle of movements;
　　identifying a set of movements during a second cycle which defines a second cycle of movements; and
　　comparing the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle.
　　The method according to claim 1, further comprising:
　　performing time wrapping on the first cycle of movements and the second cycle of movements.
　　The method according to claim 2, wherein the step of performing time wrapping includes Dynamic Time Warping (DTW) to find a warping path that describes best matching between the first cycle and the second cycle.
　　The method according to claim 2, wherein the step of performing time wrapping includes Canonical Time Warping (CTW) to align the first cycle and the second cycle under rigid registration of the feature space.
　　The method according to claim 2, wherein the step of performing time wrapping includes Longest Common Subsequence to find similar parts of two sequences.
　　The method according to any one of claims 2 to 5, wherein the step of performing time wrapping includes calculating a distance matrix and a warping path.
　　The method according to claim 6, further comprising determining whether or not there is a sideslip between the first cycle and the second cycle in response to the calculation of the warping path.
　　The method according to claim 7, wherein the step of comparing the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle includes determining there is abnormal behaviour during the second cycle when it is determined that there is the sideslip between the first cycle and the second cycle.
　　The method according to any one of the preceding claims, when it is determined that there is abnormal behaviour during the second cycle, the method further comprising representing the abnormal behavior to a user device.
　　An apparatus for determining abnormal behaviour during a cycle, the apparatus comprising:
　　at least one processor; and
　　at least one memory including computer program code; wherein
　　the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to:
　　identify a set of movements during a first cycle which defines a first cycle of movements;
　　identify a set of movements during a second cycle which defines a second cycle of movements; and
　　compare the first cycle of movements and the second cycle of movement to determine if there is abnormal behaviour during the second cycle.
　　The apparatus according to claim 10, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to perform time wrapping on the first cycle of movements and the second cycle of movements.
　　The apparatus according to claim 11, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to perform Dynamic Time Warping (DTW) to find a warping path that describes best matching between the first cycle and the second cycle.
　　The apparatus according to claim 11, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to perform Canonical Time Warping (CTW) to align the first cycle and the second cycle under rigid registration of the feature space.
　　The apparatus according to claim 11, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to perform Longest Common Subsequence to find similar parts of two sequences.
　　The apparatus according to any one of claims 11 to 14, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to calculate a distance matrix and a warping path.
　　The apparatus according to claim 15, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to determine whether or not there is a sideslip between the first cycle and the second cycle in response to the calculation of the warping path.
　　The apparatus according to claim 16, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to determine if there is abnormal behaviour during the second cycle includes determining there is abnormal behaviour during the second cycle when it is determined that there is the sideslip between the first cycle and the second cycle.
　　The apparatus according to any one of claims 11 to 16, wherein the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to represent the abnormal behavior to a user device.