JP6330093B1 - Motion determination program, motion determination device, motion determination image generation program, motion determination image generation device - Google Patents

Abstract

Information capable of accurately determining an operation of a wearable device wearer from an acceleration signal is generated. Based on acceleration signals in X, Y, and Z directions from wearable device D, the frequency of appearance of at least two of the XY, YZ, and ZX components of the acceleration signal at a predetermined time interval is determined. An appearance frequency generation unit 741 to be generated, and an RGB data generation unit 742 that generates RGB data in which the appearance frequencies of at least two components generated by the appearance frequency generation unit 741 are values of at least two colors of RGB color signals. And a label giving unit 75 for giving a label representing the wearer's movement of the wearable device D to the RGB data at a time corresponding to this, and based on the characteristics of a plurality of RGB data to which a common label is attached. And a determination unit 76 that determines whether or not the data is an operation corresponding to a label. [Selection] Figure 4

Description

  The present invention relates to a motion determination program, a motion determination device, a motion determination image generation program, and a motion determination image generation device for determining the motion of a wearable device wearer.

  2. Description of the Related Art Conventionally, it has been required to automatically determine the actions of each individual in order to bring the behavior of individuals or groups into a target state. For example, a technique has been proposed in which each individual wears a wearable device including an acceleration sensor and analyzes the waveform of an acceleration signal acquired from each acceleration sensor to determine the operation of each individual. Examples of such waveform analysis methods include waveform mapping that is determined based on how much the waveforms match, and frequency decomposition that is determined based on how much the frequency components included in the waveforms match.

JP 2006-209468 A

  However, even in a common operation, the movements shown by the individuals vary, and even an operation including repetition of the same work does not indicate the same change in acceleration value. In particular, the value and direction of acceleration shown during a certain time interval vary greatly depending on the change in speed, the order in which work is performed, and the like.

  For example, in the operation of reading a barcode with a sensor (hereinafter referred to as a barcode reader), the movement of bringing the barcode reader closer to the barcode results in a different acceleration waveform each time even if the same person repeatedly performs the same operation. Does not show similar waveforms that can be specified as Furthermore, if an abnormal movement that is not related to the original work occurs, such as scratching the head in the middle of the work, it is not possible to extract a common element by removing a portion corresponding to the abnormal movement from the waveform.

  The present invention provides a motion determination program, a motion determination device, a motion determination image generation program, and a motion determination image generation device capable of generating information capable of accurately determining the motion of a wearable device wearer from an acceleration signal. It is in.

  In order to achieve the above object, the operation determination program of the present invention causes a computer to store the acceleration signal at a predetermined time interval based on the acceleration signals in the X, Y, and Z axes from the wearable device. Of the XY components, the YZ component, and the ZX component, an appearance frequency generation process that generates an appearance frequency of at least two components, and an appearance frequency of at least two components generated by the appearance frequency generation process are expressed as RGB color signals. RGB data generation processing for generating RGB data having values of at least two colors, and label application processing for applying a label for identifying the wearer device wearer's operation to the RGB data at the predetermined time interval; The wearable device corresponding to the RGB data is based on a plurality of RGB data with a common label. A determination processing operation of the chair of the wearer, thereby to execute.

  Based on a sampling value obtained by sampling the acceleration signal at a predetermined rate in the computer, a quantization process for generating acceleration values in the X, Y, and Z axis directions per unit time, and at least based on the acceleration value, Appearance frequency generation processing for generating the appearance frequency of the two components may be executed.

  You may make the said computer perform the setting process which sets the said predetermined time interval. The setting process may set the predetermined time interval based on a detection signal detected by an external detection device. The setting process may hierarchically set a plurality of the predetermined time intervals having different lengths. A value of one color of the RGB color signals may be a value based on a detection signal detected by an external detection device.

  The present invention can also be understood as an operation determination device that realizes the functions of the above-described processes. The present invention also provides an operation determination image generation program that causes a computer to execute the appearance frequency generation process and the RGB data generation process, and an operation determination image generation apparatus that includes the appearance frequency generation unit and the RGB data generation unit. It can also be captured.

  According to the present invention as described above, it is possible to generate information that can accurately determine the operation of the wearer wearer from the acceleration signal.

Overall configuration diagram showing an operation determination system in an embodiment Explanatory drawing showing a wearable device and barcode reader used for barcode inspection A block diagram showing a wearable device in an embodiment The block diagram which shows the operation | movement determination apparatus in embodiment Graph showing acceleration signal waveform Explanatory diagram showing the direction and range of acceleration values Explanatory diagram showing the appearance frequency map Explanatory drawing showing RGB data Explanatory drawing showing the wearer who performs barcode inspection Explanatory drawing showing the time interval of barcode inspection Explanatory diagram showing multi-layer time intervals Explanatory drawing showing adjustment of time interval Explanatory diagram showing conversion of resolution of RGB data Explanatory drawing which shows the aspect of label assignment The flowchart which shows the image generation process for operation | movement determination in embodiment The flowchart which shows the learning processing in the embodiment Flowchart showing determination processing in the embodiment Explanatory drawing which shows the classification example of RGB data Explanatory drawing showing an example of shifting the interval of time intervals

[Constitution]
[Overview]
A mode for carrying out the present invention (hereinafter referred to as “the present embodiment”) will be specifically described with reference to the drawings. Note that the present embodiment is applied to an operation determination system that determines an operation of a wearable device wearer based on an acceleration signal.

  The action refers to the state of the wearer indicated by the wearer's movement. Even if the movement is stopped or the movement is in a stationary state, since there is an operation corresponding to movement or movement before and after that, and there is a movement detected by the acceleration signal, the state can be identified. Included in the action. That is, not only work and exercise, but also stop, rest, rest, sleep, posture, tone, work efficiency, concentration, fatigue, skill, etc. are included in the operation. That is, the operation referred to in this embodiment includes the concept of the wearer's state, and the operation can be read as a state, and the operation determination can be read as a state determination. The wearing may be a state that can follow the movement of the wearer to such an extent that the movement of the wearer can be specified. For this reason, not only the case where it is in direct contact with the wearer, but also clothes and members may be interposed between the wearer and the wearer.

  Determining the operation means indicating whether or not the operation is a specific operation. This includes the case where the probability is a specific action. It also includes a case where a specific action is indicated when the probability exceeds a threshold value.

  As illustrated in FIG. 1, the operation determination system S includes wearable devices D1 to Dn (hereinafter simply referred to as wearable devices D), an operation determination device M, and an access point P. The wearable device D is an information communication terminal capable of transmitting / receiving information to / from the operation determination apparatus M via the network N. The wearable device D has a main unit U and a mounting part B that is mounted on the human body. As the wearable device D, for example, a wristwatch type information communication terminal is used. In this case, the main unit U incorporates a computer including a processing unit described later in a thin casing. As shown in FIG. 2, the mounting portion B is a band that is wound around the arm and stopped.

  The operation determination apparatus M is an apparatus that can transmit and receive information to and from the wearable device D via the network N. The operation determination device M collects information from each wearable device D, and determines the operation of the wearer based on this information. The information from the wearable device D includes information input from an external device. For example, as shown in FIG. 1 and FIG. 2, a reading signal from a barcode reader R connected to the wearable device D by short-range power-saving radio such as Bluetooth (registered trademark) is also received from the wearable device D. include.

  The access point P is a relay device that relays communication between the network N, the wearable device D, and the management device M1. The access point P is also a transmission device that transmits position information. The access point P is typically a WiFi wireless LAN access point. Each access point P is registered with identification information and position information regarding the installation location, and the transmitted beacon includes identification information and information regarding the installation location. For example, the installed facility is divided into a plurality of areas, and different location information is given to each access point P installed in each area.

  The wearable device D, the operation determination device M, and the access point P are mainly configured by a computer controlled by a program. The computer is configured by a processor such as a CPU and a storage medium such as a memory, but various implementations of hardware and programs can be changed.

  The operation determination device M can be configured by a server device. Various settings, arithmetic expressions, parameters, and the like necessary for processing of each unit described later are stored in advance in a built-in or removable storage medium. A storage unit to be described later can be configured as a part of the storage area of such a storage medium.

  In the following description, a functional block diagram in which each function is illustrated as a block is used. The present invention can also be understood as a computer program for executing each function and a computer-readable storage medium storing such a program.

[Wearable device]
As described above, the wearable device D is a device that the subject of movement wears on a part of the body. As shown in FIG. 3, the wearable device D includes a communication unit 1, a current position determination unit 2, an information input unit 3, an information conversion unit 4, an information output unit 5, and a notification information output unit 6.

(Communication Department)
The communication unit 1 is a processing unit that transmits and receives information to and from the outside. The communication unit 1 can transmit / receive information to / from the operation determination device M and other wearable devices D via the network N. The network N may have any of many-to-one, many-to-many, and one-to-one relationships. For example, the communication unit 1 has a connection function between the management device M1, the management server M2, and the wearable device D by a wireless LAN such as WiFi, WiFi Direct, or the like. The communication unit 1 receives position information included in a beacon transmitted by the access point P. This reception process is called a beacon scan, and the timing for performing the beacon scan is called a scan timing. Note that, as described above, the communication unit 1 also receives information from an external input device such as the barcode reader R in the wearable device D.

(Current position determination unit)
The current position determination unit 2 is a processing unit that determines the position where the wearable device D exists, that is, the position of the staff, based on the position information from the access point P received by the communication unit 1. The current position determination unit 2 includes a signal detection unit 21 and a position determination unit 22.

  The signal detection unit 21 is a processing unit that detects position information included in a beacon received from the access point P. This position information is information for identifying a predetermined area where each access point P is installed, for example. The position determination unit 22 is a processing unit that determines the position of the wearable device D based on the position information. That is, the position determination unit 22 determines the position of the holder who wears the wearable device D. Various methods can be applied to the position detection based on the position information of the access point P. When beacons are received from access points P in a plurality of areas, the determination may be made in consideration of other information such as radio wave intensity.

(Information input part)
The information input unit 3 is a processing unit that inputs information about the wearer of the wearable device D. The information input unit 3 includes an acceleration sensor 31, a geomagnetic sensor 32, a touch sensor 33, a pulse sensor 34, a temperature sensor 35, an atmospheric pressure sensor 36, a gyro sensor 37, a camera 38, and a microphone 39.

  The acceleration sensor 31 is a sensor that senses inclination and vibration and converts it into an electrical signal corresponding to the degree of inclination and vibration. The acceleration sensor 31 is a sensor that can detect accelerations in three axial directions, ie, an X-axis direction, a Y-axis direction, and a Z-axis direction that are orthogonal to each other. The geomagnetic sensor 32 is a sensor that senses the magnitude and direction of a magnetic field and converts it into an electrical signal corresponding to the direction.

  The touch sensor 33 is a sensor that senses contact with the surface. The touch sensor 33 is a means for inputting information through contact with the owner. The touch sensor 33 is, for example, a touch panel configured in the display unit 62 described later, and enables input operations such as tap and slide. A tap is a short touch of the surface with a finger, and a slide is a touch while moving the surface with a finger. However, the tap only indicates an operation with a short time of touching the surface, and the slide merely indicates an operation with a long time of touching the surface, and may be another name such as flick or swipe.

  The pulse sensor 34 is a sensor that contacts the owner and detects the pulse of the owner. The pulse sensor 34 is also a means for inputting information through contact with the owner. As the pulse sensor 34, for example, a sensor that detects a pulse by irradiation and reflection of ultraviolet rays can be used.

  The temperature sensor 35 is a sensor that contacts an owner and converts it into an electrical signal corresponding to the body temperature of the owner. The temperature sensor 35 is also a means for inputting information in contact with the owner. The atmospheric pressure sensor 36 is a sensor that detects atmospheric pressure. For example, a barometric sensor using a piezoelectric element can be applied. The gyro sensor 37 is a sensor that detects angular velocities in the X, Y, and Z axial directions.

  The camera 38 is a device that captures still images or moving images and inputs them as image data and moving image data. The microphone 39 is an input device that converts ambient sound into an electrical signal.

(Information conversion unit, information output unit)
The information conversion unit 4 is a processing unit that converts the information input by the information input unit 3 into a format suitable for processing by the motion determination apparatus M. Information conversion includes detection information of acceleration, vibration, direction, pulse, body temperature, angular velocity, etc. of signals from the acceleration sensor 31, the geomagnetic sensor 32, the pulse sensor 34, the temperature sensor 35, the atmospheric pressure sensor 36, and the gyro sensor 37. Including conversion to parameters according to. The information conversion includes conversion of a signal input from the touch sensor 33 into a pulse signal.

  The information conversion includes a process of converting the information into parameters indicating the posture of the holder, the degree of opening / closing of the heel, the facial expression, and the like based on the image data and moving image data captured by the camera 38. Further, the information conversion includes a process of converting the signal input from the microphone 39 into a parameter indicating the volume of sound.

  The information conversion unit 4 does not need to convert all the information input from the information input unit 3. For example, the information input from the information input unit 3 may be output to the information output unit 5 as it is, and the communication unit 1 may transmit this to the operation determination device M. In this case, the motion determination apparatus M includes an information conversion unit that converts information as described above.

  The information output unit 5 is a processing unit that outputs information from the information input unit 3 together with identification information and position information of each wearable device D, and causes the communication unit 1 to transmit the information. Based on the identification output by the information output unit 5, an administrator who uses the operation determination device M that has received the information can grasp the operation, position, and other conditions of the owner of each wearable device D.

(Notification information output part)
The notification information output unit 6 is a processing unit that outputs the notification information received from the operation determination device M in a manner that can be determined by the user. The notification information output unit 6 includes a vibration unit 61 and a display unit 62. The vibration unit 61 is means for transmitting notification information to the owner through contact with the owner. As the vibration unit 61, for example, a vibrator that generates vibration using a motor as a drive source can be used. The vibration unit 61 can output not only continuous vibration but also intermittent vibration according to the pulse signal according to the notification information.

  The display unit 62 is a display that displays an image visually determined by the user, such as a still image, a moving image, and text. The display unit 62 can display a screen interface for input by the touch sensor 33.

  The wearable device D includes a storage unit that stores information necessary for the processing of each unit, although not shown. Information stored in the storage unit includes information input, processed, and output in each of the above-described units.

[Operation determination device]
The operation determination device M is a device that determines the operation of the wearer of each wearable device D based on information from each wearable device D as described above. As illustrated in FIG. 4, the motion determination device M includes a communication unit 71, an input unit 72, an acceleration value generation unit 73, an image generation unit 74, a label addition unit 75, a determination unit 76, a notification information generation unit 77, and a display unit 78. Have

[Communication Department]
The communication unit 71 is a processing unit that transmits / receives information to / from each wearable device D via the network N. The communication unit 71 functions as a reception unit that receives information and position information from the wearable device D and a transmission unit that transmits information notified to the wearable device D.

[Input section]
The input unit 72 is a processing unit that inputs various types of information necessary for the motion determination apparatus M. The input unit 72 includes a touch panel, a keyboard, a mouse, and the like. The touch panel includes one configured in a display unit 78 described later.

[Acceleration value generator]
The acceleration value generation unit 73 is a processing unit that generates acceleration values in the X, Y, and Z axis directions based on the acceleration signal from the acceleration sensor 31 of each wearable device D. The acceleration signal is continuously received from each wearable device D via the communication unit 71. As shown in FIG. 5, the acceleration signal has a waveform that frequently changes in a short time in the X-axis direction, the Y-axis direction, and the Z-axis direction. The acceleration value generation unit 73 includes a sampling unit 731 and a quantization unit 732. The sampling unit 731 is a processing unit that obtains a sampling value by sampling the acceleration signal at a predetermined rate.

The quantization unit 732 is a processing unit that generates acceleration values in the X, Y, and Z axis directions per unit time based on the sampling values obtained by the sampling unit 731. For example, the measurement value per unit time is obtained by averaging the sampling values at 2 Hz. That is, the sampling values obtained in 0.5 seconds are added and divided by the number of times. Thereby, a measurement value twice per second is obtained. For example, as shown in FIG. 6, the measured value is in a range of −12 to 12 [m / s ² ]. The quantization unit 732 converts the measured values −12 to 12 into acceleration values of 32 steps (0 to 31). That is, when the measured value is a, the acceleration value is a value obtained by subtracting 1 from the integer value of (a + 12) /0.75. 0.75 is (12 + 12) / 32. When the measurement value a = 0, the center value is 15. When the measured values in the X, Y, and Z axis directions are (0, 9.8, 0), the acceleration value is (15, 28, 15).

[Image generator]
The image generation unit 74 is a processing unit that generates an image for motion determination. The image generation unit 74 can also be regarded as an operation determination image generation apparatus and an operation determination image generation program. The image generation unit 74 includes an appearance frequency generation unit 741, an RGB data generation unit 742, and a time interval setting unit 743.

(Appearance frequency generator)
The appearance frequency generation unit 741 is a processing unit that generates the appearance frequencies of the XY component, the YZ component, and the ZX component at a predetermined time interval based on the acceleration value. The appearance frequency is the number of times each component has appeared in the coordinates. The appearance frequency generation unit 741 includes a count unit 741a and a map generation unit 741b. The counting unit 741a is a processing unit that counts the number of times each component has appeared in coordinates. For example, in the case of the acceleration value (15, 28, 15), the count unit 741a has the XY component once at the coordinates (15, 28), the YZ component once at the coordinates (28, 15), and the ZY component at the coordinates ( 15 and 15) are counted once each.

  The map generation unit 741b is a processing unit that maps the appearance frequency counted by the counting unit 741a onto predetermined coordinates. For example, if the predetermined time interval is 10 seconds, the appearance frequency is counted 20 times in 10 seconds, and the number of appearances for each of the XY component, YZ component, and ZX component on the 32 × 32 coordinates as shown in FIG. Create a recorded map. That is, a map of 32 × 32 coordinates with X as the horizontal axis and Y as the vertical axis, a map of 32 × 32 coordinates with Y as the horizontal axis and Z as the vertical axis, Z as the horizontal axis, and X as the vertical axis The number of appearances is recorded for each coordinate of the 32 × 32 coordinate map.

(RGB data generator)
The RGB data generation unit 742 is a processing unit that generates RGB data in which the appearance frequencies of the XY component, the YZ component, and the ZX component recorded by the appearance frequency generation unit 741 are values of three colors of the RGB color signals. is there. The RGB data generation unit 742 first normalizes the appearance frequency at each coordinate of the XY component map, the YZ component map, and the ZY component map to a value of 0 to 255 indicating the luminance of each color of R, G, and B. To do. As a result, as shown in FIG. 8, three RGB image data in which R, G, and B values for one pixel (pixel) are determined are generated. Further, the RGB data generation unit 742 generates RGB image data in which three pieces of image data are combined into one sheet.

(Time interval setting part)
The time interval setting unit 743 is a processing unit that sets a predetermined time interval. The predetermined time interval is a time interval at which the appearance frequency is generated by the appearance frequency generation unit 741 in order to generate one piece of RGB data. Since the contents of the acceleration signal differ depending on the setting of this time interval, the meaning indicated by the generated RGB data differs. The setting of the time interval includes the following modes.

(1) Setting of basic time interval The time interval can be set based on information obtained from the outside or the inside of the wearable device D and the operation determination device M. This is a relatively rough time interval setting and a rough time interval setting. For example, a time zone for performing the operation determination is set in advance, and the start time of the time zone in which the time obtained from the watch of the wearable device D or the clock of the operation determination device M is set in advance is set as the start time of the time interval in advance. The end point of the set time zone may be set as the end point of the time interval.

  When a different work is performed for each work table in a factory or the like, the time when the position has changed is determined based on the position of the wearer D wearer determined by the current position determination unit 22 of the wearable device D. It may be the time of the interval separation, that is, the start time or the end time. In addition, when the posture of the wearer wearing the wearable device D can be determined from the radio wave intensity when receiving the beacons from the access points P in a plurality of areas, the time when the posture has changed It is good also as an interval delimiter.

  A case where the acceleration signal from the acceleration sensor 31 exceeds a preset threshold value or is equal to or less than the threshold value may be used as a time interval. In addition, when the number of steps calculated by the information conversion unit 4 based on the acceleration signal from the acceleration sensor 31 can be received, the wearer is moving on foot, and the change in the number of steps within a predetermined time exceeds the threshold value. Alternatively, the time interval or less may be set as a time interval or less. Further, when moving in a place with a large difference in elevation, the time interval may be set when the atmospheric pressure signal from the atmospheric pressure sensor 36 exceeds a preset threshold value or is equal to or less than the threshold value.

  When the operation during sleep is targeted, information indicating the wearer's state during sleep, such as the time zone, the posture estimated by the acceleration sensor 31, and the heart rate from the pulse sensor 34, is set in advance. Whether or not this information is applicable may be used as a time interval. For example, in addition to sleep, time intervals for life such as movement, desk work, and meals can be set by this method.

  The timing at which information is externally input to the wearable device D may be used as a time interval delimiter. For example, as shown in FIG. 9 and FIG. 10, “goods out” for picking up products from the box, “inspection” for checking whether two barcodes attached to the products match, and storing the products in the box In the case of barcode inspection for performing “packing”, the product is extracted before reading by the barcode reader R, and the inspection operation is started when the reading data by the barcode reader R is first input to the wearable device D. The timing input to the wearable device D for the second time can be the end of the inspection work, and the subsequent can be the packing work.

  The input of information serving as a time interval delimiter is not limited to a specific mode. For example, the input timing of information from the touch sensor 33 may be set as a time interval, or the input timing of information from a server, a cloud, or the like via a network may be set as a time interval.

  Note that RGB data can also be used for setting such time intervals. Among the RGB data as described above, RGB data indicating a specific operation is set in advance as RGB data with coarse quantization, that is, with low resolution. Then, the RGB data generated based on the received acceleration signal is set to the same low resolution RGB data. The preset RGB data is compared with the generated RGB data, that is, the difference between the R, G, and B data is taken, and if the difference is less than or equal to the preset threshold value, the preset RGB data It can be determined that there was an action corresponding to the data. If it is larger than the threshold value, it can be determined that the operation does not correspond to preset RGB data.

  For this reason, a point in time when it can be determined that there is an operation corresponding to preset RGB data and a point in time when it is determined that the operation does not correspond to preset RGB data can be used as a time interval delimiter. In this case, since the comparison is a low resolution and a simple difference is taken, it is not always an accurate operation determination, but it can be used for setting a rough time interval.

  It should be noted that the setting of the time interval as described above may be a part of the entire time of the operation in order to determine a specific operation included in a part of the entire operation. For example, the time interval for generating RGB data for only the inspection that is a part of the barcode inspection operation may be set according to the reading timing of the barcode reader.

(2) Hierarchical time interval setting As described above, a basic time interval setting is subdivided, or a certain relatively long time interval is subdivided into short time intervals. Also good. In this case, in the case of an operation in which a periodic repetitive operation or a similar operation is performed a plurality of times, it is easy to extract the characteristics of the operation by setting the time interval relatively long. However, when the time interval is set to be long, it is difficult to appropriately adjust the start and end timings to include the feature. On the other hand, in the case of an operation that can be divided into fine work or an operation that requires detailed analysis, the time interval may be set to be relatively short. However, if the time interval is set short, there is a possibility that the characteristics of the RGB data are not stable and the characteristics are not included.

  In order to satisfy such conflicting demands, it is preferable to set hierarchical time intervals having different lengths as shown in FIG. Based on a common sampling value, acceleration values and appearance frequencies can be obtained at different time intervals, and one piece of RGB data can be generated for each time interval.

  The optimal time interval varies depending on the characteristics of the operation, the purpose of the determination, the required determination accuracy, and the like. For example, when an operation can be decomposed into a combination of motion sets of A, B, C, D, and E, RGB data is preferably obtained in each of A to E at time interval T4. Such a time interval is not always determined by the type of operation, and individual differences also occur.

  As a method for obtaining the optimum time interval, for example, as shown in FIG. 11, RGB generated at respective time intervals T1 to T4 while dividing a relatively long time interval T1 in the order of T2 → T3 → T4. An extraction process and a determination process described later are performed on the data, and a time interval at which the accuracy does not fall below the threshold value is selected. FIG. 11 shows an example in which a time interval T1 of 30 minutes is set hierarchically as a time interval T2 of 300 seconds, a time interval T3 of 60 seconds, and a time interval T4 of 10 seconds. In the figure? Indicates a time interval corresponding to RGB data whose accuracy does not exceed the threshold value.

  A time interval suitable for determination can be obtained by trying determination using RGB data obtained at each time interval and examining which time interval is suitable for determination. For example, compared to T1, T2, T3, and T4 have a larger number of RGB data, and thus there is a possibility that the accuracy is increased. However, as shown in FIG. 12, the time interval of each layer is not fixed, and is adjusted by changing it when accuracy is not improved. For example, the start and end of the time interval T1 are adjusted in accordance with the determination using the RGB data obtained at the time interval T4 so that the determination of A to E can be made with an accuracy not lower than the threshold value.

An example of disassembling the operation when the operation is subdivided, that is, when the operation of a certain superordinate concept includes a plurality of subordinate concept operations, is as follows. These operations can be further classified according to tone, concentration level, skill level, and the like.
・ Inspection work (product delivery, inspection, packing)
・ Cleaning of hotel rooms (vacuum cleaner, wiping, changing sheets)
・ Tennis (serve, receive, move, strike back)
・ Marathon (Start, Checkpoint, Mid, End)

  However, the switching of these operations does not always coincide with the time interval, such as A = product output, B = inspection, and C = packaging. Each operation may be decomposed, that is, each operation may have a relationship corresponding to a combination of time intervals shorter than the time for each operation. For example, there may be a relationship of product output → A + B + α, inspection → C−β, and packing → D + E−γ. Here, α, β, and γ are different time lengths. In addition, in FIG. 11, as a time interval in which an operation cannot be specified has occurred, movement other than the target operation or a boundary between two target operations is included in one RGB data. It may be difficult to determine movement.

  When many RGB data whose accuracy does not exceed the threshold value appears, the resolution of the RGB data is lowered and the difference value between the generated RGB data is calculated in the same manner as the above basic time interval setting. Then, by performing clustering using the difference as a distance, the data is divided into RGB data used for motion determination and RGB data not used for motion determination. That is, as shown in FIG. 13, a plurality of pixels are grouped to obtain an average value of R, G, and B values. Then, the R, G, and B values of all the pixels in the group are set as the obtained average value. Thereby, RGB data having a coarser resolution than the original RGB data is generated. Then, the difference value of the generated RGB data is calculated, and the RGB data whose difference value is equal to or less than the threshold value and the RGB data whose difference value exceeds the threshold value are classified, and the difference value is equal to or less than the threshold value. The RGB data is RGB data used for operation determination as data whose characteristics can be easily determined.

[Labeling section]
The label assigning unit 75 is a processing unit that assigns a label representing the wearer's operation of the wearable device D to RGB data at a time interval corresponding to the label. The label is identification information for specifying an operation. As shown in FIG. 14, the label to be given is information such as 1, 2, 3, A, B, and C that cannot be understood by a human at first glance. In order to allow the human to understand the meaning of the action corresponding to each label for display on the display unit 78 or as notification information, a table in which the name of the action is associated with each label is prepared. Just keep it. This name can be input from the input unit 72.

  The labels are prepared in a plurality of stages corresponding to the time intervals of the plurality of layers. For example, a label of a result for specifying the operation of the superordinate concept is assigned to the RGB data generated at the time interval of the T1 layer. The RGB data generated at the time interval of the T4 layer is given a label of the result, together with a label of the action element for specifying the action of the lower concept included in the action of the higher concept, that is, the action element. Note that the label does not need to be fixed, and can be adjusted by changing it when the accuracy does not improve, or when it matches the operation corresponding to another label.

[Determining part]
The determination unit 76 is a processing unit that determines the wearer's operation of the wearable device corresponding to the RGB data based on a plurality of RGB data to which a common label is assigned. The determination of the operation is performed based on the common characteristics of the plurality of RGB data. As the determination unit 76, for example, a deep learning tool using a convolutional neural network can be used. Regarding the temporal continuity of the change in acceleration, the RGB data is expressed as spatial continuity, so that it is compatible with the convolution process.

  The determination unit 76 includes an extraction unit 761 and an identification unit 762. The extraction unit 761 is a processing unit that extracts common features of RGB data to which a common label is assigned. The feature is a converted image obtained by converting a partial image of each RGB data, and is hereinafter referred to as a feature amount. Such feature amount extraction is, for example, a convolution filter that extracts the shade of each pixel of the partial image by a filter of a predetermined size and determines each pixel value of the converted image, and the position on the coordinates is Even if they are shifted, it can be performed by repeating a process such as pooling that is regarded as the same.

  The identification unit 762 is a processing unit that identifies whether or not the RGB data is RGB data corresponding to a label. This identification corresponds to, for example, a label from a final total connection layer, starting from a total connection layer in which feature values obtained from a plurality of RGB data are combined as nodes (neurons), through a plurality of total connection layers. This is done by outputting the probability of being RGB data and the probability of not being RGB data. The probability of the RGB data corresponding to the label is the accuracy, and the probability of having and not having it is 100% in total. The output of each fully connected layer is a calculation result of a predetermined parameter assigned to each fully connected layer and the output of the previous fully connected layer.

  In the above process, by substituting an output value less than zero with zero, the extracted feature amount can be further emphasized, and the processing amount can be reduced. Further, by partially disconnecting all the coupling layers, it is possible to prevent over-learning that is optimized only for learning data.

[Notification information generator]
The notification information generation unit 77 is a processing unit that generates notification information to be transmitted from the motion determination device M to the wearable device D. The notification information is information notified to the wearer of wearable device D, and includes determination accuracy, label, and instruction information. A wearer of wearable device D can determine whether or not a correct determination is made with respect to his / her operation based on the determination accuracy and the label. The instruction information is information that prompts the wearer of the wearable device D to perform a desired action so as to be in a target state. Generation and transmission of such notification information can be instructed by input from the input unit 72.

[Display section]
The display unit 78 is a processing unit that displays various types of information necessary for the operation determination apparatus M. The display unit 78 includes, for example, an acceleration signal received by the communication unit 71, an acceleration value generated by the acceleration value generation unit 73, an appearance frequency generated by the appearance frequency generation unit 741, and a map thereof, and RGB generated by the RGB data generation unit 742. The data, the time interval set by the time interval setting unit 743, the label provided by the label applying unit 75, the determination result by the determining unit 76, the determination accuracy, and the like are displayed. The display unit 78 can display a screen interface for input by the input unit 72. An administrator who refers to the determination result, determination accuracy, and the like displayed on the display unit 78 can make adjustments by changing the time interval and the label using the input unit 72.

  Although not shown, the operation determination apparatus M also includes a storage unit that stores information necessary for the processing of each unit described above. Information stored in the storage unit includes acceleration signal, acceleration data, appearance frequency, appearance frequency map, RGB data, label, label and name table, determination result, determination accuracy, and the like. The information stored in the storage unit includes information set in advance, various settings such as threshold values, and the like.

[Processing procedure]
An example of the processing of the present embodiment as described above will be described with reference to the flowcharts of FIGS. 15 to 17 in addition to the above drawings. Note that a processing method according to the following procedure is also an embodiment of the present invention.

(Generation of motion judgment images)
First, the procedure for generating an RGB image for operation determination will be described with reference to the flowchart of FIG. The acceleration signal from each wearable device D is received by the communication unit 71 and input to the acceleration value generation unit 73 (step 101). The sampling unit 731 of the acceleration value generation unit 73 obtains a sampling value obtained by sampling the input acceleration signal, and the quantization unit 732 converts the sampling value into respective acceleration values in the X, Y, and Z axis directions (steps). 102).

  The counting unit 741a of the appearance frequency generation unit 741 counts the appearance frequency of the XY component, the YZ component, and the ZY component at a predetermined time interval, and the map generation unit 741b appears on the coordinates of the XY component, the YZ component, and the ZY component. A map in which the number of times is recorded is created (step 103).

  The RGB data generation unit 742 normalizes the appearance frequency at each coordinate of the XY component map, the YZ component map, and the ZY component map to a value indicating the luminance of each color of R, G, and B, thereby Three RGB image data in which R, G, and B values for the pixels are determined are generated, and further, RGB data in which the three image data are combined into one sheet is generated (step 104).

(Learning)
Next, a processing procedure for learning the operation determination based on the RGB data generated as described above will be described with reference to the flowchart of FIG. A large amount of RGB data is collected and stored in the storage unit. For example, about 100 to 10,000 sheets are collected. The label attaching unit 75 assigns a label to the collected RGB data for each time interval hierarchy (step 201). The labeling is performed after classification according to the input from the input unit 72, for example. The RGB data to which labels are assigned in this way is called learning data.

  Based on the learning data, the extraction unit 761 of the determination unit 76 extracts a common feature amount of RGB data to which a common label is assigned (step 202), and whether or not the identification unit 762 is RGB data corresponding to a label. The identification result is output with accuracy (step 203). This process is performed for all learning data (step 204). All learning data is identified (YES in step 204). If the accuracy is equal to or lower than a preset threshold value (NO in step 204), it is determined that adjustment is necessary, such as time intervals (see FIG. 12). ) Is adjusted (step 206).

  This adjustment includes operations such as changing labels and excluding RGB data whose accuracy is equal to or less than a threshold value from the determination target. In the adjustment, the time interval setting unit 743 and the determination unit 76 autonomously change the time interval by a preset increment or decrement, exclude RGB data whose accuracy is equal to or less than the threshold from the learning data, and the like. Alternatively, it may be performed by inputting an instruction from the input unit 72. It is conceivable that the autonomous labeling by the labeling unit 75 is performed on the label of the motion element.

  For example, with emphasis on real-time characteristics, the upper RGB data with a long time interval in the hierarchical structure of time intervals is compared with the accuracy of the lower RGB data with a short time interval, and the lower RGB data is used as much as possible. Also good. Further, it is not always necessary to select a time interval with high accuracy. A time interval having a certain length may be set as a minimum time interval by excluding those that are too divided and have a short time interval.

  After such adjustment, recognition by learning data is performed (steps 202, 203, and 204). If the accuracy exceeds the threshold value (YES in step 205), the learning is terminated.

(Operation judgment)
A determination process for determining whether or not the operation corresponds to the label based on the acceleration signal from the wearable device D after learning as described above will be described with reference to the flowchart of FIG. First, the process of generating the acceleration value, the appearance frequency, and the RGB data based on the input acceleration signal is the same as the above steps 101 to 104 (steps 301 to 304). Then, the determination unit 75 determines whether or not the generated RGB data is an operation corresponding to the label (step 305). This determination result is also output with accuracy. If the newly determined RGB data also has an accuracy exceeding the threshold value, a label is added and included in the learning data. The learning based on the newly added RGB data may be performed at a time different from the actual determination process.

  It should be noted that even when an abnormal operation that is not related to or necessary for the original operation is included, it is preferable that it can be labeled and identified. For example, for a low-occurrence movement such as picking up an object, hitting it, or being interrupted, we want to check for mistakes, so we can label the action and use it for anomaly detection.

  From the above, RGB data can be classified as shown in FIG. First, RGB data that can be used to determine normal operation includes RGB data that can be used to determine detailed operating elements, and RGB data that can be used to determine operating elements is used to determine the operation according to the purpose. RGB data that can be used is included. That is, the RGB data that can recognize the normal operation includes RGB data that is not subject to the operation determination. In addition to the RGB data that can determine the normal operation, there is RGB data that can determine the abnormal operation as described above.

  As described above, notification information may be output to the wearable device D when the operation is determined. For example, when it is determined from the RGB data that the operator's movement is “fatigue”, or when the label indicating the fatigue level is a label that should be “fatigue”, the notification information generation unit 77 generates Send instruction information. As the instruction information, information that prompts breaks, meals, other work, and rearrangement of a plurality of types of time patterns is set in advance. As a result of transmitting these instruction information, when the operation of the operator is determined by the operation determination device M, it can be determined whether or not the target state has been reached by the instruction information. For example, it can be seen that the condition of the worker has been improved, for example, because it is no longer determined as fatigue or the label indicating the degree of fatigue is no longer a label that should be “fatigue”. By performing this multiple times, it is possible to determine notification information effective for improving the worker's state. That is, notification information that is effective for the purpose can be determined, which can be used for improving the state.

  In addition, when the wearable device D wearer is a marathon runner, based on the RGB data taken at the time of practice and the result of motion determination based on the data, pace distribution guides, correct form guides, etc. are used as instruction information. Send. Thereby, the coaching environment which leads to a better result is realizable. This is effective for all sports and work.

[Function and effect]
(1) The operation determination program according to the present embodiment causes the computer to transmit the XY component, the YZ component, and the ZX of the acceleration signal at predetermined time intervals based on the acceleration signals in the X, Y, and Z axes from the wearable device D. Appearance frequency generation processing for generating the appearance frequency of at least two components of the components, and RGB data in which the appearance frequency of at least two components generated by the appearance frequency processing is a value of at least two of the RGB color signals RGB data generation processing for generating a label, label addition processing for adding a label for identifying the wearer device wearer's operation to RGB data at a predetermined time interval, and a plurality of RGB data with a common label And a determination process for determining the wearer's operation of the wearable device D corresponding to the RGB data based on That.

  In addition, the motion determination apparatus according to the present embodiment is based on the acceleration signals in the three-axis directions of X, Y, and Z from the wearable device D, and among the XY component, YZ component, and ZX component of the acceleration signal at a predetermined time interval, Appearance frequency generation unit 741 that generates the appearance frequency of at least two components, and RGB data in which the appearance frequency of at least two components generated by the appearance frequency generation unit 741 is a value of at least two of the RGB color signals An RGB data generation unit 742 to generate, a label giving unit 75 for giving a label representing the wearer's operation of the wearable device D to RGB data at a time corresponding to this, and a plurality of RGB data with a common label And a determination unit 76 that determines whether or not the RGB data is an operation corresponding to a label based on the above feature.

  In addition, the operation determination image generation program of the present embodiment causes a computer to execute the above-described appearance frequency generation processing and RGB data generation processing. In addition, the image generation device for operation determination according to the present embodiment includes the appearance frequency generation unit 741 and the RGB data generation unit 742 described above.

  Thus, in this embodiment, the influence of instability in the time axis direction can be suppressed by using the appearance frequency of the acceleration component at a predetermined time interval. In the case of waveform analysis, collation is performed in a state in which the waveform and frequency components change in the direction of the time axis, so if the order of work changes or there are individual differences in the speed and movement of work It is very difficult to determine the same operation. This is the same because a mode of change in the time axis direction is included even when the waveform is mapped. However, in the present embodiment, for example, even if the operations included in the same operation are performed in different orders within a predetermined time interval, the appearance frequencies are likely to be the same, and therefore can be determined as the same operation. In addition, for example, even if there are individual differences in work speed and movement magnitude, the appearance frequency is likely to show the same mode, so that it can be determined as the same action.

  In the present embodiment, data suitable for determination based on image data can be generated by using the appearance frequency as RGB data. However, as image data for recognition by a convolutional neural network, there is a method of recognizing an operation using image data obtained by imaging an operation target. However, when recognizing an operation from captured image data, since it is necessary to use a large amount of image data captured along the time axis of the operation, the necessary image data becomes enormous and the processing load increases. Also, as with waveform analysis or more, there are changes in the order of work, and individual differences in work speed and motion magnitude, so the recognition accuracy will not improve or the image data required for improvement will be enormous. Become. That is, it can be said that it has been common technical knowledge that there is a problem in using not only the acceleration signal but also the image data in order to recognize the motion.

  On the other hand, as a result of intensive studies, the inventor of the present invention uses an acceleration signal for the determination of the operation contrary to the above-mentioned technical common sense, but is not a simple acceleration signal, its XY component, YZ component, It came to pay attention to the appearance frequency of a ZX component. Then, instead of mapping the acceleration signal, etc., by generating RGB data using simple numerical information such as the appearance frequency as the value of each pixel, the increase in the amount of data along the time axis can be suppressed and accurate. We have found that accurate judgment can be realized. In other words, the idea of using an appearance frequency that has not been noticed in the past has been recalled, and further, the idea of intentionally converting it to RGB data has been reached.

  Since the RGB data generated in this manner has an appearance frequency of zero because there are zero appearance frequencies compared to captured image data and the like, the RGB data becomes a substantial compressed file. Since the data size is reduced, the storage capacity for storing data can be reduced, and the transfer speed can be increased. In addition, the amount of calculation required for convergence of learning can be suppressed. In addition, since the RGB data is visualized, a certain amount of pattern can be recognized even when viewed by a human. Such an advantage cannot be obtained by simply imaging the acceleration value as it is or simply imaging the numerical value obtained by calculating the acceleration value.

(2) The motion determination program according to the present embodiment is a quantization process for generating acceleration values in the X, Y, and Z axis directions per unit time based on a sampling value obtained by sampling an acceleration signal at a predetermined rate. And an appearance frequency generation process for generating an appearance frequency of at least the two components based on the acceleration value.

  In addition, the motion determination apparatus of the present embodiment includes a quantization unit 732 that generates acceleration values of X, Y, and Z axis components per unit time based on a sampling value obtained by sampling an acceleration signal at a predetermined rate, and an acceleration. And an appearance frequency generation unit 741 that generates the appearance frequencies of the two components based on the values.

  For this reason, since the acceleration signal is smoothed by quantization, a fine high-frequency component is cut, and an influence of a motion error, a wearing deviation of the wearable device D, and the like can be suppressed. Furthermore, since it is quantized, the data is simplified, and the amount of calculation at the time of arithmetic processing is suppressed.

(3) The operation determination program of the present embodiment causes a computer to execute a setting process for setting a predetermined time interval. For this reason, it is possible to make an accurate determination by adjusting the time interval according to the operation determination result.

(4) The operation determination program of this embodiment sets a predetermined time interval based on a detection signal detected by an external detection device. For this reason, for example, as in barcode inspection, it is easy to match the operation interval and the time interval interval, and an accurate determination can be realized.

(5) The operation determination program of the present embodiment hierarchically sets a plurality of predetermined time intervals having different lengths. For this reason, by trying the determination at various time intervals, it is possible to set an optimal time interval for accurate operation determination.

[Modification]
The present invention is not limited to the above embodiment.
(1) When converting the appearance frequency into RGB data, there is a problem of the timing of the time interval separation. For example, when the assembly process is A, B, or C, the timing of the separation of one operation is the same as that of the other operation. It may not match the timing of the break. In this case, if the timing of the time interval is the same in each layer, the difference between the timing of the operation and the timing of the time interval is constant even if the time interval of acquiring the RGB data is changed. Since it does not change, there is a possibility that accuracy does not increase.

  In order to cope with this, as shown in FIG. 19, the time intervals for conversion to RGB data may be overlapped. In other words, the time interval set hierarchically does not require that the upper layer time interval delimiter and the lower layer time interval delimiter match. By shifting the separation of the time intervals, the time intervals of the upper layer and the lower layer can be overlapped. As in the case of the adjustment by changing the time interval, the determination accuracy is checked while changing the overlap, that is, the amount of deviation, and an appropriate amount of deviation is found.

  As a result, compared to the case where each layer break is set at a common timing, there is a possibility that RGB data in which the time interval break and the operation break coincide with each other, or there is RGB data that does not include measurement data of different operations. The possibility can be increased.

  Moreover, since various RGB data can be obtained by overlapping, there is a high possibility that highly accurate RGB data is generated, and the accuracy of determination by deep learning or the like can be increased. Further, as in the case of shortening the time interval, it is possible to determine the change in the situation at an early stage.

(2) As data at the time of learning, the appearance frequency may be extracted by assigning three components of XY component, YZ component, and ZY component to any of R, G, and B, respectively. Get higher. However, if the appearance frequency used for conversion into RGB data is at least two of the XY component, YZ component, and ZY component, the operation can be determined. For this reason, two components such as the XY component and the YZ component may be two values of R and G, and the other value of B may be another value related to the operation. That is, the value of one color among the RGB color signals may be a parameter value of a type different from the appearance frequency. For example, the amount of change in the acceleration values Z and X may be used, or the angular velocity value from the gyro sensor 37 may be used. As a result, in the case of an operation in which a parameter value of a type different from the appearance frequency exhibits a remarkable feature, the determination accuracy is increased.

(3) The wearer may be a moving object. In addition to humans, moving bodies such as animals and cars, robots, and the like may be used. However, the present invention has an advantage that the operation can be determined from the complicated and diverse movements of human beings. The attachment to the body by the attachment part only needs to be performed to such an extent that an acceleration signal that can specify the movement is obtained. In addition to the arm, it may be worn at any position on the body such as fingers, head, neck, chest, waist, and feet. For this reason, the mounting portion may also be an adhesive tape, a double-sided adhesive sheet, a hook-and-loop fastener, etc., in addition to a band, belt, string, glasses, earplugs, and the like. Clothing, hats, supporters, accessories, footwear, and the like may be used as the mounting portion. The main unit may not touch the skin directly. That is, the contact for information transmission may be direct to the skin or indirect with a part of the wearing part interposed.

DESCRIPTION OF SYMBOLS 1 Communication part 2 Current position determination part 21 Signal detection part 22 Position determination part 3 Information input part 31 Acceleration sensor 32 Geomagnetic sensor 33 Touch sensor 34 Pulse sensor 35 Temperature sensor 36 Pressure sensor 37 Gyro sensor 38 Camera 39 Microphone 4 Information conversion part 5 Information output unit 6 Notification information output unit 61 Vibration unit 62 Display unit 71 Communication unit 72 Input unit 73 Acceleration value generation unit 74 Image generation unit 741 Appearance frequency generation unit 741a Count unit 741b Map generation unit 742 RGB data generation unit 743 Time interval setting Unit 75 label adding unit 76 determining unit 761 extracting unit 762 identifying unit 77 notification information generating unit 78 display unit D wearable device P access point R barcode reader M operation determining device N network S operation determining system

Claims (10)

On the computer,
Appearance frequency that generates the appearance frequency of at least two components of the XY component, the YZ component, and the ZX component of the acceleration signal in a predetermined time interval based on the X, Y, and Z-axis acceleration signals from the wearable device Generation process,
RGB data generation processing for generating RGB data in which the appearance frequencies of at least two components generated by the appearance frequency generation processing are values of at least two colors of RGB color signals;
A labeling process for assigning a label for identifying the wearer's wearer's action to the RGB data at the predetermined time interval;
Based on a plurality of RGB data with a common label, a determination process for determining the wearer's operation of the wearable device corresponding to the RGB data;
An operation determination program characterized by causing In the computer,
Quantization processing for generating acceleration values in the X, Y, and Z axis directions per unit time based on a sampling value obtained by sampling the acceleration signal at a predetermined rate;
An appearance frequency generation process for generating an appearance frequency of at least the two components based on the acceleration value;
The operation determination program according to claim 1, wherein:   The operation determination program according to claim 1, wherein the computer is caused to execute a setting process for setting the predetermined time interval.   The operation determination program according to claim 3, wherein the setting process sets the predetermined time interval based on a detection signal detected by an external detection device.   5. The operation determination program according to claim 3, wherein the setting process sets a plurality of the predetermined time intervals having different lengths in a hierarchical manner.   6. The operation determination program according to claim 1, wherein a value of one color of the RGB color signals is set to a parameter value different from the appearance frequency. Appearance frequency that generates the appearance frequency of at least two components of the XY component, the YZ component, and the ZX component of the acceleration signal in a predetermined time interval based on the X, Y, and Z-axis acceleration signals from the wearable device A generator,
An RGB data generation unit that generates RGB data in which the appearance frequency of at least two components generated by the appearance frequency generation unit is a value of at least two of the RGB color signals;
A label giving unit for giving a label representing an operation of the wearer of the wearable device to the RGB data at a time corresponding to the label;
A determination unit that determines whether each RGB data is an operation corresponding to the label based on characteristics of a plurality of RGB data with a common label;
An operation determination device characterized by comprising: A quantization unit that quantifies acceleration in the X, Y, and Z axis directions per unit time based on a sampling value obtained by sampling the acceleration signal at a predetermined rate;
A calculation unit for calculating the appearance frequency of the two components based on the acceleration value;
The operation determination apparatus according to claim 7, further comprising: On the computer,
Appearance frequency that generates the appearance frequency of at least two components of the XY component, the YZ component, and the ZX component of the acceleration signal in a predetermined time interval based on the X, Y, and Z-axis acceleration signals from the wearable device Generation process,
RGB data generation processing for generating RGB data in which the appearance frequencies of at least two components generated by the appearance frequency generation unit are values of at least two colors of RGB color signals;
A program for generating an image for motion determination, characterized in that Appearance frequency that generates the appearance frequency of at least two components of the XY component, the YZ component, and the ZX component of the acceleration signal in a predetermined time interval based on the X, Y, and Z-axis acceleration signals from the wearable device A generator,
An RGB data generation unit that generates RGB data in which the appearance frequency of at least two components generated by the appearance frequency generation unit is a value of at least two of the RGB color signals;
An image generation apparatus for operation determination characterized by comprising: