JP2022087045A - State determination system, state determination method, and state determination program - Google Patents

Abstract

To perform damage inspection based on objective indices with stable accuracy.SOLUTION: A state determination system S includes a master-side actuator 52, a slave-side actuator 62, a motion control unit 111, a determination unit 315 or 114, and a notification unit 214. The motion control unit 111 performs motion control to control a grip mechanism 65 to grip an object 4 to be inspected by driving the slave-side actuator 62 in accordance with user operation input to an operation mechanism 55, and to control the operation mechanism 55 to transmit a reactive force to the user by driving the master-side actuator 52 in accordance with the reactive force from the object 4 to be inspected input to the grip mechanism 65. The determination unit 315 or 114 makes a determination on a damage of the object 4 to be inspected, on the basis of control parameters used by the motion control unit 111 in the motion control.SELECTED DRAWING: Figure 1

Description

The present invention relates to a state determination system, a state determination method, and a state determination program.

Conventionally, various articles have been inspected for damage. For example, by inspecting packaging containers for packaging the contents of foods and the like for damage, air leaks in the packaging containers can be detected before the product reaches the consumer, and the quality of the product can be improved. It becomes possible to guarantee.

Patent Document 1 discloses an example of a technique for inspection of such a packaging container or the like. In the technique disclosed in Patent Document 1, various sensors such as a sensor for measuring the weight and temperature and a sensor for measuring the inclusion of foreign matter by electromagnetic waves are used for the packaging container after packaging the contents. The inspection can be performed from the viewpoint of.

Japanese Unexamined Patent Publication No. 2011-253469

However, in order to perform the inspection as disclosed in Patent Document 1, not only various sensors but also large-scale equipment such as a conveyor for inspection is required. In this respect, it is relatively easy to install large-scale equipment for each product in the inspection at the shipping stage in factories and the like. On the other hand, it is not realistic to install large-scale equipment in inspections at distribution centers, retail stores, etc., where it is necessary to detect damage to various products during transportation.

Under these circumstances, at distribution centers and the like, manual inspections are carried out at the timing of sorting and picking. However, in the manual inspection, the inspection accuracy depends on the personal feeling (that is, the subjectivity of the worker). As described above, when the inspection is performed depending on the subjectivity of the worker, it is difficult to objectively explain the basis of the judgment in the inspection. Further, in the inspection depending on the subjectivity of the worker, the inspection accuracy may not be stable due to factors such as the difference in the skill level of each worker and the physical condition of the worker. In this way, if the inspection accuracy is not stable, the inspection result will be incorrect, and the packaging container will be delivered to the consumer even though it is damaged, and finally, the complaint from the consumer. Will occur.

The present invention has been made in view of such a situation. An object of the present invention is to realize an inspection with stable accuracy based on an objective index when inspecting for damage.

In order to solve the above problems, the state determination system according to the embodiment of the present invention is
The actuator on the operation mechanism side that operates the operation mechanism, and
The actuator on the gripping mechanism side that operates the gripping mechanism,
By driving the gripping mechanism side actuator in response to the user's operation input to the operating mechanism, the gripping mechanism controls the operation of gripping the inspection target, and the inspection target input to the gripping mechanism. An operation control means for performing an operation control in which the operation mechanism controls an operation of transmitting the reaction force to the user by driving the actuator on the operation mechanism side in response to a reaction force from an object.
A determination means for determining damage to the inspection object based on the control parameters used by the operation control means in the operation control, and a determination means.
A notification means for notifying the user of the determination result of the determination means, and
It is characterized by having.

According to the present invention, when inspecting for damage, it is possible to realize an inspection with stable accuracy based on an objective index.

It is a block diagram which shows an example of the whole structure of the state determination system which concerns on one Embodiment of this invention. It is a schematic diagram which shows the concept of the basic principle of control of the operation of the controlled object apparatus. It is a schematic diagram which shows the concept of control when a force / tactile transmission function is defined. It is a schematic diagram which shows the concept of a master-slave system. It is a schematic diagram which shows the example of the information which is stored as the extraction result of an operation. It is a schematic diagram which shows the concept of control of a force / tactile transmission function using scaling in a frequency domain. It is a schematic diagram which shows the basic structure of the state determination apparatus. It is a block diagram which shows an example of the hardware and the functional block of a control unit for realizing the state determination process. It is a block diagram which shows an example of the hardware and the functional block of a wearable terminal for realizing the state determination process. It is a block diagram which shows an example of the hardware and the functional block of a server device for realizing the state determination process. It is a flowchart explaining the flow of the pre-processing in the state determination processing. It is a flowchart explaining the flow of the gripping start processing in the state determination processing. It is a flowchart explaining the flow of the determination process in the state determination process. It is a schematic diagram which shows the notification example when it is determined that the inspection object is not damaged. It is a schematic diagram which shows the notification example when it is determined that the inspection object is damaged. It is a flowchart explaining the flow of the gripping end processing in the state determination processing. It is a block diagram which shows an example of the hardware and the functional block of a control unit in 2nd Embodiment. It is a block diagram which shows an example of the hardware and the functional block of a wearable terminal in 2nd Embodiment. It is a flowchart explaining the flow of the pre-processing in the machine learning processing in the 2nd Embodiment. It is a flowchart explaining the flow of the construction process in the machine learning process in the 2nd Embodiment. It is a flowchart explaining the flow of the pre-processing in the state determination processing in 2nd Embodiment. It is a flowchart explaining the flow of the determination process in the state determination process in 2nd Embodiment.

Hereinafter, each of the first embodiment and the second embodiment, which are examples of the embodiments of the present invention, will be described with reference to the accompanying drawings.

<First Embodiment>
[System configuration]
FIG. 1 is a block diagram showing an overall configuration of a state determination system S according to the present embodiment. As shown in FIG. 1, the state determination system S includes a state determination device 1, a wearable terminal 2, and a server device 3. In addition, the user U, the network N, and the inspection object 4 are also shown in the figure.

The state determination device 1, the wearable terminal 2, and the server device 3 are connected to each other so as to be communicable with each other. Communication between these devices may be performed in accordance with any communication method, and the communication method is not particularly limited. Further, the communication between each of these devices may be wireless communication, wired communication, or a combination of wired communication and wireless communication. In addition, communication between each of these devices may be performed directly between the devices, or may be performed via a network N or a relay device (not shown). The network N is realized by, for example, a network such as a LAN (Local Area Network), the Internet, or a mobile telephone network.

The state determination device 1 is a device that assists the user U in inspecting the inspection target 4 for damage. The state determination device 1 realizes a function as a gripping device for gripping the inspection object 4 based on the operation of the user U, and the inspection object 4 is based on the control parameters related to the force and tactile sensation used in the gripping. Determines if is damaged.
In the following, as an example for explanation, it is assumed that the state determination device 1 is realized by a robot manipulator capable of a remote operation (that is, teleoperation) by the user U. More specifically, it is assumed that the state determination device 1 includes a gripping mechanism for gripping the inspection object 4 and is a portable reacher (that is, a magic hand) that can be carried and used by the user U. do. In order to realize the function as the reacher, the state determination device 1 includes an operation mechanism 55 that accepts the operation of the user U, a master side unit 50 for operating the operation mechanism 55 as a master device, and an inspection object 4. It includes a gripping mechanism 65 for gripping, a slave-side unit 60 for operating the gripping mechanism 65 as a slave device, and a control unit 10 for integrally controlling these operations.

In the state determination device 1, the master side unit 50 and the slave side unit 60 communicate with each other via the control unit 10, and the operation control unit included in the control unit 10 controls the operation. Operates as a master device (here, a device that accepts the operation of the user U), and the other operates as a slave device (here, a device that grips the inspection object 4 as a robot manipulator). In this case, the user U realizes an operation from a vicinity or a remote operation by operating the master device while observing the operation of the slave device directly or by moving images or the like. Further, at the time of operation by this user, the control unit 10 included in the state determination device 1 controls the operation, so that the operation of the master device (here, the operation by the user U accepted by the operation mechanism 55) is transmitted to the slave device. At the same time, a function (that is, a bilateral control function) of feeding back the input of the reaction force from the object to the slave device (here, the reaction force from the inspection object 4 with respect to the grip of the grip mechanism 65) to the master device is realized. To.

Here, the specific configuration of the operation mechanism 55 is not particularly limited, but for example, an operation accompanied by a force-tactile sensation such as grasping, sliding (that is, sliding), twisting, or pushing by the hand of the user U or the like is performed. It can be realized by the receiving mechanism. Similarly, the specific configuration of the gripping mechanism 65 is not particularly limited, but for example, the member that grips the inspection object 4 opens and closes in response to the operation accompanied by the force-tactile sensation of the user U as described above. It can be realized by a mechanism for gripping with a force-tactile sensation such as rotation or ascending / descending.
However, the configurations of the operation mechanism 55 and the grip mechanism 65 are merely examples, and the configuration is such that the force tactile sensation in the operation of the user U and the force tactile sensation in the grip can be mutually transmitted between the master device and the slave device as control parameters. All you need is. That is, the specific configurations of the operating mechanism 55 and the gripping mechanism 65 are not particularly limited to the above-mentioned examples.

The wearable terminal 2 provides the user U with various information for assisting the inspection regarding the damage of the inspection object 4 (for example, the determination result by the state determination device 1, the work instruction corresponding to the determination result, etc.). It is a device that notifies the user interface and the like. The wearable terminal 2 is realized by, for example, a head-mounted display (HMD: Head Mounted Display) that is mounted on the head of the user U and used. In such a head-mounted display, the user U is notified by using a technology such as virtual reality (VR), augmented reality (AR), or mixed reality (MR). Is possible. For example, when using augmented reality technology, the actual scenery that can be seen by the user U through the transmissive display and the virtual reality displayed on the transmissive display (for example, virtually generated images and texts, etc.) Can be superimposed to notify the user U.

Further, the notification by the wearable terminal 2 includes not only the display on the display included in the wearable terminal 2 but also the output of the sound (voice, warning sound, etc.) from the speaker included in the wearable terminal 2 and the wearable terminal 2. It can also be realized by blinking the warning light.
It should be noted that these processes by the wearable terminal 2 are realized by the wearable terminal 2, the state determination device 1, and the server device 3 communicating with each other and cooperating with each other.

The server device 3 is a server that manages information used for processing performed by the state determination device 1 and the wearable terminal 2 and identifies the inspection object 4 by image recognition. note that. In the figure, the server device 3 is shown as a single server device, but for example, the server device 3 may be realized by a plurality of server devices in the form of a cloud server or the like.

The inspection object 4 is an article to be inspected for damage by the user U. The inspection object 4 is not particularly limited, but as an example for explanation below, the inspection object 4 is a packaging container for packaging food (for example, sweets such as potato chips) and the like as contents. Suppose. Here, in such a packaging container, carbon dioxide or nitrogen gas is filled as a content instead of oxygen in order to prevent deterioration (for example, oxidation) or damage of food or the like. In the present embodiment, by inspecting such a packaging container as an inspection object 4 for damage, leakage of carbon dioxide or nitrogen gas (so-called air leakage), which is the content, can be detected, or food or the like can be fluid. In such a case, it is possible to detect leakage of food or the like which is this fluid. As a result, deterioration of the contents and outflow of the contents due to air leakage can be detected before the product reaches the consumer, and the quality of the product can be guaranteed.

Next, the outline of the process for assisting the inspection regarding the damage of the inspection object 4 by the state determination system S having such a configuration will be described.
First, the state determination system S controls the operation of the state determination device 1 based on the operation of the user U (here, control by the bilateral control function), so that the state determination device 1 grips the inspection object 4. .. Further, the state determination system S acquires control parameters related to force and tactile sensation used in the control of the state determination device 1. Further, the state determination system S determines the damage of the inspection object based on the acquired control parameters related to the force and tactile sensation. Then, the state determination system S notifies the user of the determination result of the determination unit 114.

As described above, the state determination system S objectively determines the damage of the inspection object 4 based on the control parameters related to the force and tactile sensation when the inspection object 4 is gripped by the state determination device 1. Therefore, the state determination system S can visualize the work by converting the sensation in the work (here, the force and tactile sensation) into data. As a result, the inspection accuracy does not depend on the personal feeling (that is, the subjectivity of the operator), and the inspection can be performed with stable accuracy. Further, since it has such a configuration, the state determination system S can be easily realized without requiring large-scale equipment such as various sensors and a conveyor for inspection. Therefore, it is possible to apply various products to inspections performed at distribution centers, retail stores, etc. for finding damage during transportation.
That is, according to the state determination system S, when inspecting for damage, it is possible to realize an inspection with stable accuracy based on an objective index.

[Operation control for controlled device]
Next, as a premise of the description of the specific processing in the above-mentioned state determination system S, the controlled target device (here, the master side unit 50, the operation mechanism 55, and the slave provided in the state determination device 1) in the present embodiment. The basic principle of operation control for the side unit 60 and the gripping mechanism 65) will be described.

It should be noted that human movements (that is, human physical actions) are composed of individual "functions" such as one joint alone or in combination.
Therefore, hereinafter, in the present embodiment, the "motion" refers to an integrated function realized by using individual "functions" of parts in the human body as components. For example, an operation that involves an operation on the state determination device 1 (for example, an operation that causes the operation mechanism 55 included in the state determination device 1 to be operated by a hand) includes fingers and wrists of the hand, and joints of arms and shoulders connected to these. It is an integrated function whose components are the functions of.

(Basic principle)
The basic principle of motion control in this embodiment is defined by transformation and inverse transformation because any motion can be mathematically expressed by three elements of force source, velocity (position) source, and transformation representing motion. By supplying control energy to the controlled system from the ideal force source and ideal velocity (position) source that are in a dual relationship with respect to the variable group, the extracted motion is structured, reconstructed or expanded and amplified, and the motion is reversible. It is automatically realized (reproduced).

FIG. 2 is a schematic diagram showing the concept of the basic principle of controlling the operation of the controlled object device in the present embodiment.
The basic principle shown in FIG. 2 represents a control rule of an actuator that can be used to realize human movement (here, an actuator for operating the gripping mechanism 65 included in the state determination device 1). The operation of the actuator is determined by performing an operation in at least one region of the position (or velocity) or the force with the current position of the actuator as an input.
That is, the basic principle of controlling the operation of the controlled target device in the present embodiment is the controlled target system CS, the force / speed allocation conversion block FT for each function, the ideal power source block FC, or the ideal speed (position) source block PC. It is expressed as a control rule including at least one of the above and the inverse conversion block IFT.

The controlled object system CS is a robot operated by an actuator (here, a state determination device 1), and controls the actuator based on acceleration or the like. Here, the controlled target system CS realizes the function of one or a plurality of parts in the human body, but if the control rule for realizing the function is applied, the specific configuration is It does not necessarily have to be a form that imitates the human body. For example, the controlled object system CS can be a robot that causes a link to perform a one-dimensional sliding motion by an actuator.

The function-specific force / velocity allocation conversion block FT is a block that defines the conversion of control energy into a region of speed (position) and force set according to the function of the controlled target system CS. Specifically, in the function-specific force / velocity allocation conversion block FT, coordinate conversion is defined in which a reference value (reference value) of the function of the controlled target system CS and the current position of the actuator are input. In general, this coordinate conversion converts an input vector having a reference value and a current velocity (position) as elements into an output vector consisting of a velocity (position) for calculating a control target value of the velocity (position), and also converts the reference value. And the input vector whose element is the current force is converted into the output vector consisting of the force for calculating the control target value of the force. Specifically, the coordinate conversion in the function-specific force / velocity allocation conversion block FT is expressed in a generalized manner as in the following equations (1) and (2).

However, in the equation ( ₁ ), x'1 to x'n ( _n is an integer of 1 or more) is a velocity vector for deriving the state value of the velocity, and _x'a to x'm ( _m is 1 or more). (Integer) is a vector whose element is the velocity based on the reference value and the action of the actuator (the velocity of the mover of the actuator or the velocity of the object moved by the actuator), and h _1a to h _nm is an element of the transformation matrix representing the function. Is. Further, in the equation (2), f''1 to f''n ( _n is an integer of ₁ or more) is a force vector for deriving the state value of the force, and _f''a to _f''m (n). m is an integer of 1 or more) is a vector whose elements are the reference value and the force based on the action of the actuator (the force of the mover of the actuator or the force of the object moved by the actuator).

By setting the coordinate conversion in the function-specific force / speed allocation conversion block FT according to the function to be realized, various operations can be realized and the operation accompanied by scaling can be reproduced.
That is, in the basic principle of controlling the operation of the controlled device in the present embodiment, the system expressing the function of realizing the variable (variable in the real space) of the actuator alone in the function-specific force / velocity allocation conversion block FT. It "converts" to the entire set of variables (variables in virtual space), and allocates control energy to the control energy of velocity (position) and the control energy of force. Therefore, it is possible to independently give the control energy of the velocity (position) and the control energy of the force as compared with the case where the control is performed with the variable (variable in the real space) of the actuator alone.

The ideal force source block FC is a block that performs operations in the force region according to the coordinate transformation defined by the functional force / velocity allocation conversion conversion block FT. In the ideal force source block FC, a target value related to the force when performing a calculation based on the coordinate transformation defined by the functional force / velocity allocation conversion conversion block FT is set. This target value is set as a fixed value or a variable value depending on the function to be realized. For example, when realizing the same function as the reference value, set zero as the target value, and when scaling, set the enlarged / reduced value of the information indicating the function to be reproduced. can.

The ideal velocity (position) source block PC is a block that performs an operation in the area of velocity (position) according to the coordinate transformation defined by the function-specific force / velocity allocation conversion block FT. In the ideal speed (position) source block PC, a target value regarding the speed (position) when performing a calculation based on the coordinate conversion defined by the function-specific force / speed allocation conversion block FT is set. This target value is set as a fixed value or a variable value depending on the function to be realized. For example, when realizing the same function as the reference value, set zero as the target value, and when scaling, set the enlarged / reduced value of the information indicating the function to be reproduced. can.

The inverse conversion block IFT is a block that converts the values in the region of velocity (position) and force into the values in the region of input to the controlled system CS (for example, voltage value or current value).
Based on such a basic principle, when the position information in the actuator of the controlled target system CS is input to the function-specific force / speed allocation conversion block FT, the speed (position) and force information obtained based on the position information is obtained. In the function-specific force / velocity allocation conversion block FT, the control rules for each of the position and force regions according to the function are applied. Then, in the ideal power source block FC, the force is calculated according to the function, and in the ideal speed (position) source block PC, the speed (position) is calculated according to the function, and the force and the speed (position) are calculated. Control energy is distributed to each.

The calculation results in the ideal force source block FC and the ideal speed (position) source block PC are information indicating the control target of the controlled target system CS, and these calculation results are used as input values of the actuator in the inverse conversion block IFT for control. It is input to the target system CS.
As a result, the actuator of the controlled target system CS executes the operation according to the function defined by the function-specific force / speed allocation conversion block FT, and the target robot operation is realized.
That is, in the present embodiment, the robot (here, the state determination device 1) can more appropriately realize the human movement at the time of a predetermined action.

(Example of defined function)
Next, a specific example of the function defined by the force / speed allocation conversion block FT for each function will be described.
In the function-specific force / velocity allocation conversion block FT, the speed (position) obtained based on the current position of the input actuator and the coordinate conversion targeting the force (conversion from real space to virtual space corresponding to the realized function). ) Is defined.
In the function-specific force / velocity allocation conversion block FT, the velocity (position) and force from the current position and the velocity (position) and force as the reference value of the function are input, and the velocity (position) and force are respectively. The control law of is applied in the acceleration dimension.
That is, the force in the actuator is represented by the product of mass and acceleration, and the velocity (position) in the actuator is represented by the integral of acceleration. Therefore, by controlling the velocity (position) and the force through the region of acceleration, the current position of the actuator can be acquired and the desired function can be realized.

Specific examples of various functions will be described below.
(Force / tactile transmission function)
FIG. 3 is a schematic diagram showing the concept of control when the force / tactile transmission function is defined in the function-specific force / speed allocation conversion block FT. Further, FIG. 4 shows a master device (here, the master side unit 50 and the operation mechanism 55 included in the state determination device 1) and a slave device (here, the state determination device 1) to which the force / tactile transmission function is applied. It is a schematic diagram which shows the concept of the master-slave system including the slave side unit 60, and the gripping mechanism 65) provided with the state determination apparatus 1.

As shown in FIG. 4, as a function defined by the force / speed allocation conversion block FT for each function, the operation of the master device is transmitted to the slave device, and the reaction from the object (for example, the inspection object 4) to the slave device is performed. It is possible to realize a function (bilateral control function) of feeding back the force input to the master device.
In this case, the coordinate conversion in the function-specific force / velocity allocation conversion block FT is expressed by the following equations (3) and (4).

However, in the equation (3), _x'p is the velocity for deriving the state value of the velocity (position), and _x'f is the velocity related to the state value of the force. Further, _x'm is the speed of the reference value (input from the master device) (differential value of the current position of the master device), and x's is the current _speed of the slave device (differential value of the current position). Further, in the equation (4), f _p is a force related to the state value of the velocity (position), and f _f is a force for deriving the state value of the force. Further, f _m is the force of the reference value (input from the master device), and f _s is the current force of the slave device.

(Action extraction function)
Next, the extraction function of human motion realized by this embodiment will be described in detail.
As described above as (basic principle), a robot having a predetermined mechanism (for example, a robot having a mechanism corresponding to the function of the human body) is configured, and human movements are made to follow this robot. By detecting the time-series movements of this robot, it is possible to extract human movements.

For example, in the case of the state determination device 1 as shown in FIG. 1, a plurality of actuators included in the state determination device 1 (see FIG. 7 described later) following the operation of a human moving the hand to operate the state determination device 1. (Illustrated.) The position of each actuator when each moves is detected by a plurality of position sensors (illustrated in FIG. 7 described later) in chronological order, and these positions are stored in a storage device. ..
In this case, instead of the detection result of the position of each actuator, each value (for example, calculated in time series) for deriving the state value obtained as the coordinate conversion result in the force / velocity allocation conversion conversion block FT for each function. Each value on the left side of the equation (4) may be stored in the storage device.

FIG. 5 is a schematic diagram showing an example of information stored as an operation extraction result, and FIG. 5A shows a case where the time-series positions of a plurality of actuators (here, actuators A1 to A4) are stored. FIG. 5B shows a case where the coordinate conversion result of the time-series equation (4) is stored.
Referring to FIG. 5A, for example, the actuator A1 stores a time-series position such as position p1 at time t1, position p2 at time t2, position p3 at time t3, and so on.
Further, referring to FIG. 5B, for example, regarding the coordinate conversion result _x''a1 , the coordinate conversion result q1 at time t1, the coordinate conversion result q2 at time t2, the coordinate conversion result q3 at time t3, and so on. The series values are stored.
As a result, if a human actually performs a predetermined action in a predetermined action only once, the robot can reproduce the motion such as learning and reproduction function of the predetermined motion without performing the predetermined motion thereafter. Can be done. In addition, the control parameters related to the force and tactile sensation used in this predetermined operation can be saved as a log.

(Scaling function)
In the above-mentioned force / tactile transmission function, the position, force and time scaling function can be further realized.
The scaling function is a function that expands or contracts the scale of the output position, force, or time with respect to the reference control. With the scaling function, for example, the magnitude of the movement of the master device can be reduced and reproduced on the slave device, the strength (force) of the movement of the master device can be increased and reproduced on the slave device, or the movement of the master device can be reproduced. It can be reproduced by a slave device by reducing the speed of. In addition, by using the scaling function for at least one of the information of the position or force stored in the storage device, for example, the magnitude of the stored movement can be reduced and reproduced by the slave device, or the strength of the stored movement can be achieved. It can be reproduced with a slave device by strengthening the memory (power).
Hereinafter, a configuration example for realizing the scaling function will be described.

(Force / tactile transmission function with scaling)
When the force / tactile transmission function accompanied by scaling is realized, the coordinate conversion in the function-specific force / velocity allocation conversion block FT in FIG. 2 is expressed by the following equations (5) and (6).

In the case of the coordinate conversion shown in the equations (5) and (6), the position of the slave device is multiplied by α (α is a positive number), the force of the slave device is multiplied by β (β is a positive number), and the master device is used. Be transmitted.
With such a scaling function, for example, when gripping, the force-tactile sensation associated with the operation of the user U can be suppressed or emphasized, so that more delicate work or work requiring more force can be performed. It will be effective if.

(Force / tactile transmission function with force limitation due to scaling)
When the force / tactile transmission function with force limitation due to scaling is realized, the coordinate conversion in the function-specific force / velocity allocation conversion block FT in FIG. 2 is expressed by, for example, the following equations (7) to (10). ..
When realizing such a function, it is appropriate to consider the following conditions.
-Continuous up to the velocity dimension (existence condition of Jacobian determinant)
-The force after restriction is a monotonic increase function of the original force (stability condition)
・ When f _s <a, f _s = f _shat or f _s ≒ f _shat (f _shat is a parameter included in the functional force / velocity allocation conversion block FT in equations (9) and (10)).
(Conditions that guarantee control performance in the safe area)
・ Saturation function (condition to realize position limit)
It is also possible to adopt the atan function as another function that satisfies these conditions.

In the case of the coordinate transformations shown in the equations (7) to (10), if the force of the slave device is less than a, the slave device and the master device can be obtained by applying the coordinate transformations of the equations (7) and (8). Is controlled by a similar force. On the other hand, when the force of the slave device is a or more, the scaling function works by applying the coordinate transformations of the equations (9) and (10) so that the slave device does not exceed the force of (1 / b + a). Be controlled.
With such a scaling function, for example, in the process according to the present embodiment, it is possible to make a determination with stable accuracy based on a change in position (that is, a movement amount) while the force in gripping is constant. .. In addition, it is possible to avoid damage to the inspection object 4. In the coordinate transformations shown in the above-mentioned (7) to (10), for example, by setting the scaling target to the position instead of the force, it is possible to realize the force / tactile transmission function with the limitation of the position. Is.

(Force / tactile transmission function using scaling in the frequency domain)
FIG. 6 is a schematic diagram showing the concept of controlling the force / tactile transmission function using scaling in the frequency domain.
In FIG. 6, the outputs of the master device and the slave device are input to the function-specific force / speed allocation conversion block FT after passing through the high-pass filter (HPF) and the low-pass filter (LPF), respectively.
In the function-specific force / speed allocation conversion block FT, coordinate conversion for the high frequency range is applied to the output of the master device and slave device that have passed the high-pass filter, and the output of the master device and slave device that has passed the low-pass filter is applied. On the other hand, the coordinate transformation for the low frequency range is applied.
That is, the function-specific force / speed allocation conversion block FT separates the inputs from the master device and the slave device into signals in the high frequency range and the low frequency range, and applies coordinate conversion corresponding to each frequency range.

As shown in FIG. 6, when the force / tactile transmission function using scaling in the frequency domain is realized, the coordinate conversion in the function-specific force / velocity allocation conversion block FT is expressed as the following equations (11) to (14). Will be done.

In the case of the coordinate conversion shown in the equations (11) to (14), in the low frequency region, by applying the coordinate transformations of the equations (11) and (12), the slave device and the master device have the same position. In the high frequency range, by applying the coordinate transformations of equations (13) and (14), the position of the slave device is α times (α is a positive number) and the force of the slave device is β times (β). Is a positive number) and transmitted to the master device.
Such a scaling function enables, for example, the slave device to emphasize the sensation when penetrating or breaking an object and transmit it to the master device.

(Reproduction of function using time scaling)
In the reproduction of the function by the state determination device 1, the target is to thin out or interpolate the information indicating the learned and stored function (for example, the time series data representing the extraction result of the action shown in FIG. 5). By setting it as a value, time scaling can be realized.
Specifically, it is stored by thinning out the information indicating the function learned and stored, and then using it as the target value of the calculation in the ideal power source block FC or the ideal speed (position) source block PC. Functions (operations) can be reproduced at high speed. Similarly, by interpolating the information indicating the learned and stored function and using it as the target value, the stored function (operation) can be reproduced at a low speed.

In this way, when reproducing the stored function (operation) at high speed, it is sufficient to perform a low-speed and accurate operation at the time of extracting the action, and it is possible to perform a high-speed and accurate operation at the time of reproduction.
Further, when the stored function (operation) is reproduced at a low speed, the operation performed at a normal speed can be reproduced slowly.

[Configuration of status determination device]
Next, the configuration of the state determination device 1 will be described with reference to FIG. 7. FIG. 7 is a schematic diagram showing the basic configuration of the state determination device 1.

As shown in FIG. 7, the state determination device 1 includes a control unit 10, a master side unit 50, an operation mechanism 55, a slave side unit 60, and a gripping mechanism 65. Further, the master side unit 50 includes a master side driver 51, a master side actuator 52, and a master side position sensor 53. Further, the master side unit 50 operates the operation mechanism 55 by the master side actuator 52. Similarly, the slave-side unit 60 includes a slave-side driver 61, a slave-side actuator 62, and a slave-side position sensor 63. Further, the slave side unit 60 operates the gripping mechanism 65 by the slave side actuator 62.
In the following description, when the master side (here, the operation mechanism side) and the slave side (here, the gripping mechanism side) are described without distinction, some of the names and symbols are omitted. They are simply referred to as "units", "drivers", "actuators", and "position sensors".

The state determination device 1 operates as a master device and a slave device by the cooperation of the control unit 10, the master side unit 50, and the slave side unit 60 as described above with reference to FIG. 1. When operating as one device of the device and the slave device, it is installed in the actuator (that is, the master side actuator 52 or the slave side actuator 62) of the unit (that is, the master side unit 50 or the slave side unit 60) that operates as the other device. The detection result of the position sensor (that is, the master side position sensor 53 or the slave side position sensor 63) is used as an input to perform an operation according to the function.
As described above, the functions implemented in the state determination device 1 can be variously changed by switching the coordinate conversion defined by the function-specific force / velocity allocation conversion block FT realized by the control unit 10. ..

The control unit 10 controls the entire state determination device 1, and is composed of a processor such as a CPU (Central Processing Unit) and an information processing device including a storage device such as a memory or a hard disk.
The control unit 10 has functions of a force / speed allocation conversion block FT for each function in FIGS. 2 and 3, an ideal power source block FC, an ideal speed (position) source block PC, and an inverse conversion block IFT. .. Then, the control unit 10 controls to operate as one of the master device and the slave device by these functions.

Therefore, the control unit 10 acquires a reference value (hereinafter, referred to as “reference value”) for each function provided in the state determination device 1. This reference value is, for example, a time-series detection value output from a position sensor installed in an actuator of a unit that operates as one of the master device and the slave device. When the time-series detection value is acquired from the unit operating as the other device to the control unit 10 as a reference value in real time, the control unit 10 can be configured by a communication interface (communication I / F). Further, in order to realize the above-mentioned (operation extraction function), when the time-series detection value of the unit operating as the other device is stored and sequentially read as a reference value and acquired by the control unit 10, the control unit is used. 10 can be configured by a storage device such as a memory or a hard disk.

That is, first, the time-series detection value detected by the position sensor of the unit operating as the other device is input to the control unit 10 as a reference value. The detected value in this time series represents the operation of the unit operating as the other device, and the control unit 10 refers to the speed (position) and force information derived from the input detected value (position). Then, apply the coordinate transformation set according to the function.

Then, the control unit 10 performs an operation in the region of the velocity (position) with respect to the velocity (position) for deriving the state value of the velocity (position) obtained by the coordinate conversion. Similarly, the control unit 10 performs an operation in the force region with respect to the force for deriving the state value of the force obtained by the coordinate transformation. Further, the control unit 10 performs dimensional unification processing such as acceleration on the calculation result in the calculated velocity (position) region and the calculation result in the force region, and is set according to the function. Apply the inverse transformation of the coordinate transformation. As a result, the control unit 10 converts the calculation result in the calculated velocity (position) region and the calculation result in the force region into a value in the region of input to the actuator.

Further, in the control unit 10, a functional block for performing a process for assisting the inspection regarding the damage of the inspection object 4 functions. This functional block will be described later with reference to FIG.

The driver converts the value in the region of the input to the actuator, which is inversely converted by the control unit 10, into a specific control command value (voltage value, current value, etc.) for the actuator, and outputs the control command value to the actuator.
The actuator is driven according to the control command value input from the driver, and controls the position of the controlled device.
The position sensor detects the position of the controlled target device controlled by the actuator and outputs the detected value to the control unit 10.

With such a configuration, the state determination device 1 transfers the velocity (position) and force obtained from the position of the actuator detected by the position sensor into the velocity (position) region and the force region by coordinate conversion according to the function. Convert to a state value. As a result, control energy is distributed to each of the velocity (position) and the force according to the function. Then, each state value is inversely converted into a control command value, and the actuator is driven by the driver according to this control command value.

Therefore, the state determination device 1 can calculate the state values of the speed (position) and the force required to realize the desired function by detecting the positions of one of the actuators of the master device and the slave device. By driving the other actuator of the master device and the slave device based on these state values, the position and force of the master device and the slave device can be controlled to the desired state.

Further, the state determination device 1 can realize different functions by switching the coordinate conversion according to the function in the control unit 10. For example, the storage device provided in the state determination device 1 stores coordinate conversions corresponding to various functions corresponding to a plurality of functions, and performs coordinate conversions according to one of the functions according to the purpose. By selecting the device, it is possible to realize various functions in the state determination device 1.

For example, when realizing the above-mentioned function as (force / tactile transmission function), the state determination device 1 inputs the reference value input to the control unit 10 in real time from the unit operating as the other device. It can be the acquired value of position and force. In this case, one device can be controlled in real time in conjunction with the operation of the unit operating as the other device. That is, in this case, since the coordinate transformation represented by the equation (2) is defined in the control unit 10, the position of the master side actuator 52 operating as the master device and the position of the slave side actuator 62 operating as the slave device. It is controlled so that the difference between the and is zero.

Further, when the above-mentioned function is realized as (force / tactile transmission function), the force / tactile sensation in the operation applied by the operator to the master side actuator 52 operating as the master device is transmitted to the slave device, and the slave operates as the slave device. The reaction force from the object (for example, the object to be inspected 4) acting on the side actuator 62 is fed back to the master side actuator 52 operating as the master device. As a result, the operation performed on the master device can be accurately reproduced by the slave device, and the reaction force from the object input to the slave device can be accurately transmitted to the master device.

In addition, for example, when the above-mentioned function is realized as (operation extraction function), the state determination device 1 is the other device in which the reference value input to the control unit 10 is acquired and stored in advance. It can be the acquisition value of the time-series position and force of the unit that operates as. In this case, the function of the state determination device 1 can be realized based on the operation of the unit that operates as the other device prepared in advance. That is, the target function can be reproduced in the state determination device 1 in a state where there is no unit that operates as the other device.

In addition, for example, when realizing the above-mentioned function as (scaling function), the state determination device 1 reduces the magnitude of the movement of the unit operating as, for example, one device by the scaling function, and the other device. It can be reproduced with a unit that operates as one device, it can be reproduced with a unit that operates as the other device by increasing the strength (force) of the movement of the unit that operates as one device, or it can be reproduced with a unit that operates as one device. It can be reproduced with a unit that operates as the other device by reducing the speed of.

In addition, for example, when both the above-mentioned function as (motion extraction function) and the above-mentioned function as (scaling function) are realized, the state determination device 1 is stored in the storage device by the motion extraction function. By using the scaling function for at least one of the information of position or force, for example, the magnitude of the stored movement can be reduced and reproduced by a unit operating as the other device, or the strength (force) of the stored movement can be reproduced. ) Can be strengthened and reproduced with a unit that operates as the other device.

As described above, the control unit 10 realizes the function as a gripping device by controlling the operation of the master side unit 50 as a master device and the operation of the slave side unit 60 as a slave device, and further. The "state determination process" is performed in cooperation with the wearable terminal 2 and the server device 3.
Here, the state determination process is a series of processes for determining whether or not the inspection object 4 is damaged in order to assist the inspection regarding the damage of the inspection object 4.

FIG. 8 is a block diagram showing an example of hardware and functional blocks of the control unit 10 for realizing this state determination process. As shown in FIG. 8, the control unit 10 includes a processor 11, a ROM 12, a RAM 13, a communication unit 14, a storage unit 15, an input unit 16, an output unit 17, and a drive 18. Further, although not shown in FIG. 8, the driver and the position sensor are connected to the control unit 10 as shown in FIG. 7. Each of these parts is connected by a signal line and sends and receives signals to and from each other.

The processor 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 15 into the RAM 13. Data and the like necessary for the processor 11 to execute various processes are also appropriately stored in the RAM 13.
In the figure, the processor 11 is shown as a single processor, but this is only an example. For example, the processor 11 may be realized by a plurality of processors. In this case, for example, the function for controlling the operation as the master device or the slave device described above (corresponding to the "operation control unit 111" or the "parameter acquisition unit 112" in the figure) and the state determination process in cooperation with this. (Corresponding to "determination standard setting unit 113" and "determination unit 114" in the figure) may be realized by different processors. Further, in this case, the processor is composed of various processing units such as a single processor, a multiprocessor, and a multicore processor, and these various processing units and an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). ) Etc. may be included. Further, in this case, the ROM 12, the RAM 13, and the like may be provided for each processor.

The communication unit 14 controls communication for the processor 11 to communicate with another device (for example, a wearable terminal 2 or a server device 3). The storage unit 15 is composed of a semiconductor memory such as a DRAM (Dynamic Random Access Memory) and stores various data.

The input unit 16 is composed of various buttons and a touch panel, or an external input device such as a mouse and a keyboard, and inputs various information according to a user's instruction operation. The output unit 17 is composed of a display, a speaker, or the like, and outputs an image, a voice, a warning sound, or the like.
A removable medium (not shown) made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 18. The program read from the removable media by the drive 18 is installed in the storage unit 15 as needed.

When the state determination process is realized in such a hardware configuration, as shown in FIG. 8, the processor 11 includes an operation control unit 111, a parameter acquisition unit 112, a determination standard setting unit 113, and a determination unit 114. , Works.
Further, in the case of realizing the state determination process in such a hardware configuration, as shown in FIG. 8, a parameter storage unit 151 and a determination reference storage unit 152 are set in one area of the storage unit 15. ..
Data necessary for realizing processing is appropriately transmitted and received between these functional blocks, even if not specifically mentioned below.

As described above, the motion control unit 111 controls the motion of the master unit 50 operating as the master device and the slave unit 60 operating as the slave device by applying the force / tactile transmission function. That is, the motion control unit 111 functions as a function-specific force / speed allocation conversion block FT, an ideal force source block FC, an ideal speed (position) source block PC, and an inverse conversion block IFT in FIGS. 2 and 3. Realize. Further, in this case, the motion control unit 111 defines the force / tactile transmission function in the function-specific force / speed allocation conversion block FT as described above with reference to FIG. 3, and applies the force / tactile transmission function. Control the operation. Further, in this case, the motion control unit 111 realizes a different function by switching the coordinate conversion according to the function.

The parameter acquisition unit 112 acquires control parameters (hereinafter, referred to as “control parameters related to force / tactile sensation”) used in the operation control to which the force / tactile transmission function applied by the motion control unit 111 is applied. In the following, as an example for explanation, it is assumed that the parameter acquisition unit 112 acquires a value indicating a position and a value indicating a force obtained from the position of the actuator detected by the position sensor as control parameters related to force and tactile sensation. do.

Here, as described above, the force in the actuator can be calculated as the product of the mass and the acceleration, and the velocity (position) in the actuator can be calculated by integrating the acceleration. Therefore, for example, the parameter acquisition unit 112 refers to the time-series position of each actuator in the above-mentioned (motion extraction function) with reference to FIG. 5 (a), and the above-mentioned (operation) with reference to FIG. 5 (b). Based on the information corresponding to the coordinate conversion result of the time-series equation (4) in the extraction function of), the value indicating the position and the value indicating the force are calculated by performing operations such as integration in real time, and these forces are calculated. Acquire control parameters related to tactile sensation.

Further, the parameter acquisition unit 112 stores the acquired control parameters related to the force and tactile sensation in the parameter storage unit 151. That is, the parameter storage unit 151 functions as a storage unit for storing control parameters related to force and tactile sensation.

The determination standard setting unit 113 sets a standard for the determination unit 114, which will be described later, to determine whether or not the inspection object 4 is damaged. In the following, as an example for explanation, it is assumed that the determination is performed based on a predetermined threshold value. Therefore, the determination standard setting unit 113 sets the predetermined threshold value and determines using the set predetermined threshold value as the determination standard. It is stored in the reference storage unit 152. That is, the determination standard storage unit 152 functions as a storage unit that stores a predetermined threshold value that is a determination standard.

Here, in the present embodiment, in order to make a more appropriate determination, the predetermined threshold value is appropriately different depending on the type of the inspection target 4 to be determined this time and the like. The specific numerical value of the predetermined threshold value according to the type and the like of the inspection object 4 to be determined this time is transmitted from the wearable terminal 2 to the determination standard setting unit 113.

The determination unit 114 determines whether or not the inspection object 4 is damaged in order to assist the inspection regarding the damage of the inspection object 4. As described above, this determination is performed based on a predetermined threshold value set by the determination standard setting unit 113 and stored in the determination standard storage unit 152. The specific determination method by the determination unit 114 will be described later with reference to the flowcharts of FIGS. 11, 12, 13, and 16.
Further, the determination unit 114 transmits the determination result to the wearable terminal 2.

[Wearable device configuration]
Next, the configuration of the wearable terminal 2 will be described with reference to FIG. FIG. 9 is a block diagram showing an example of hardware and functional blocks of the wearable terminal 2. As shown in FIG. 9, the wearable terminal 2 includes a processor 21, a ROM 22, a RAM 23, a communication unit 24, a storage unit 25, an input unit 26, an output unit 27, a camera 28, a sensor 29, and the like. It is equipped with. Each of these parts is connected by a signal line and sends and receives signals to and from each other. Of these, the functions of the parts other than the camera 28 and the sensor 29 as hardware are the same as the parts of the same name provided in the state determination device 1 described above with reference to FIG. Therefore, the duplicated description of the functions of each of these parts as hardware will be omitted. On the other hand, the camera 28 and the sensor 29 will be described below.

The camera 28 includes elements for photographing such as an optical lens and an image sensor, and peripheral circuits for controlling them. The image data generated by shooting the subject with the camera 28 or the image data indicating the surrounding environment such as for head tracking is output to the processor 21 as appropriate.

The sensor 29 measures the acceleration, the gyro (that is, the angle, the angular velocity, etc.), and the direction in the wearable terminal 2. The measurement result of the sensor 29 is also output to the processor 21 as appropriate.

The processor 21 identifies the position and orientation of the user's head based on the outputs of the cameras 28 and the sensor 29, and the transmissive display included in the output unit 27 based on the identified position and orientation of the head. Notification to the user U (here, display) is performed at an appropriate display position. Thereby, as described above, the actual scenery that can be visually recognized by the user U through the transmissive display and the virtual reality displayed on the transmissive display (for example, a virtually generated image or text) are included. ) Can be superimposed to notify the user U. That is, in the present embodiment, the technique of transmitting force and tactile sensation and the technique of mixed reality can be combined to assist (that is, support) the work and judgment in the inspection and the like of the user U.

As described above, this notification may be realized by outputting a sound (voice, warning sound, etc.) from the speaker included in the output unit 27, blinking the warning light, or the like.
Further, the processor 21 may be realized by including not only a CPU but also a processor for image processing such as a GPU (Graphics Processing Unit) in order to display such a mixed reality technique.

In such a hardware configuration, when the state determination process is realized, the instruction interpretation unit 211, the object photographing unit 212, the object information management unit 213, and the notification unit are used in the processor 21 as shown in FIG. 214 and the work information management unit 215 function.
Further, in this case, as shown in FIG. 9, an object information storage unit 251 and a work information storage unit 252 are set in one area of the storage unit 25.
Data necessary for realizing processing is appropriately transmitted and received between these functional blocks, even if not specifically mentioned below.

The instruction interpretation unit 211 interprets the content of the instruction operation from the user U. Here, the instruction operation from the user U may be accepted by a pressing operation or the like for a button or the like provided in the input unit 26, but is accepted by another method by the instruction interpretation unit 211 performing a process for interpretation. May be. The reason for doing so is that, in the present embodiment, it is assumed that the user U uses the wearable terminal 2 and at the same time carries the state determination device 1 by hand to perform an operation for grasping. In this case, the operation of pressing the button may interfere with this gripping operation. Therefore, in the wearable terminal 2, for example, the speaker included in the input unit 26 receives an instruction operation by an instruction (so-called voice command) composed of voice from the user U. Then, the instruction interpretation unit 211 interprets the content of the instruction composed of the voice. Alternatively, the camera 28 is made to accept an instruction operation consisting of gestures such as the movement of the hand of the user U. Then, the instruction interpretation unit 211 interprets the instruction operation consisting of the gesture.
In this way, by making it possible to accept an instruction consisting of voice and an instruction operation consisting of gestures, it is possible to prevent the operation for instructing the wearable terminal 2 from interfering with the operation for grasping the state determination device 1. However, the convenience of the user U can be further improved.

The object photographing unit 212 generates image data (hereinafter, referred to as “inspection object image data”) with the inspection object 4 as a subject by taking an image using the camera 28. Then, the object photographing unit 212 transmits the generated image data of the inspection object to the server device 3. The server device 3 identifies which type of inspection object 4 is the inspection object 4 which is the subject by performing image analysis of the inspection object image data.

The object information management unit 213 manages the object information. Here, the object information is information about each of the articles that can be the inspection object 4. For example, the product name, the image data of the appearance (that is, the packaging container), the image data of the identification information such as the bar code, the corresponding predetermined threshold value, and the gripping mechanism at the time of inspection of the article that can be the inspection object 4. The object information includes image data and the like of the position to be grasped in 65. It should be noted that this is an example, and in addition, general information about the product such as the manufacturer, the expiration date, and the retail price of the article that can be the inspection object 4 may be included in the object information.

The object information management unit 213 acquires the object information by communicating with the server device 3 and the server device (not shown) of the manufacturer that manufactures the inspection object 4. Further, the object information management unit 213 updates the object information so that it has the latest contents by periodically repeating such communication.
Then, the object information management unit 213 stores the object information acquired and updated in this way in the object information storage unit 251. That is, the object information storage unit 251 functions as a storage unit for storing the object information.

The notification unit 214 notifies the user U of various information for assisting the inspection regarding the damage of the inspection object 4, the user interface, and the like. The various types of information correspond to, for example, the identification result of the inspection object 4 by the image analysis by the server device 3, the determination result of whether or not the inspection object 4 is damaged by the state determination device 1, and the determination result. Work instructions, etc. Further, the user interface is, for example, a user interface related to the state determination process.
As described above, the notification by the notification unit 214 is displayed on the transmissive display provided in the output unit 27, the output of the sound (voice, warning sound, etc.) from the speaker provided in the output unit 27, and the blinking of the warning light. It is realized by such as. A specific example of the notification by the notification unit 214 will be described later with reference to FIGS. 15 and 16.

The work information management unit 215 manages the work information. Here, the work information is information on the work to be performed in relation to the inspection object 4. For example, the time zone for inspecting whether the inspection object 4 is damaged or not, and the location of the sorting destination according to the result of whether or not it is damaged (for example, the location of the shelf or containment vessel to be the sorting destination). The work information includes the time zone for sorting and picking accompanied by such inspection. Further, the work information may include the identification information of the user U who actually performed the work such as inspection, the time zone when the work was actually performed, and the work history such as the inspection result. It should be noted that this is only an example, and in addition, for example, the work information includes information on the physical distribution of the inspection object 4, such as the delivery address, the time zone to be delivered, and the number of items to be delivered. May be good.

The work information management unit 215 acquires work information by communicating with the server device 3 and the server device (not shown) of the logistics company that delivers the inspection object 4. Further, the work information management unit 215 updates the work information so that it has the latest contents by periodically repeating such communication.
Then, the work information management unit 215 stores the work information acquired and updated in this way in the work information storage unit 252. That is, the work information storage unit 252 functions as a storage unit for storing work information.

[Server device configuration]
Next, the configuration of the server device 3 will be described with reference to FIG. FIG. 10 is a block diagram showing an example of hardware and functional blocks of the server device 3. As shown in FIG. 10, the server device 3 includes a processor 31, a ROM 32, a RAM 33, a communication unit 34, a storage unit 35, an input unit 36, an output unit 37, and a drive 38. .. Each of these parts is connected by a signal line and sends and receives signals to and from each other. The functions of these parts as hardware are the same as the parts having the same name provided in the state determination device 1 described above with reference to FIG. 8 and the parts having the same name provided with the wearable terminal 2 described above with reference to FIG. be. Therefore, the duplicated description of the functions of each of these parts as hardware will be omitted.

In such a hardware configuration, when the state determination process is realized, the object information management unit 311, the identification unit 312, and the work information management unit 313 function in the processor 31 as shown in FIG. ..
Further, in this case, as shown in FIG. 10, an object information management unit 351 and a work information storage unit 352 are set in one area of the storage unit 35.
Data necessary for realizing processing is appropriately transmitted and received between these functional blocks, even if not specifically mentioned below.

The object information management unit 311 manages the object information in the same manner as the object information management unit 213 included in the wearable terminal 2. Here, since the details of the object information are described above in the explanation of the object information management unit 213, the description thereof will be omitted again here. Further, the object information management unit 311 is also the same as the object information management unit 213, but communicates with the object information management unit 213 and the server device (not shown) of the manufacturer of the inspection object 4. By doing, the object information is acquired and updated.
Then, the object information management unit 311 stores the object information acquired and updated in this way in the object information management unit 351. That is, the object information management unit 351 functions as a storage unit for storing the object information.

The identification unit 312 identifies which type of inspection object 4 is the inspection object 4 which is the subject by performing image analysis of the inspection object image data. Here, as described above, the inspection object image data is image data with the inspection object 4 as the subject, which is generated by the object photographing unit 212 of the wearable terminal 2 taking a picture using the camera 28. be. The identification unit 312 realizes this identification by image analysis using artificial intelligence (AI: Artificial Intelligence) or a predetermined algorithm.

For example, the identification unit 312 contains image data of the inspection object, image data of the appearance (that is, packaging container) of each article that can be the inspection object 4 included in the object information, and information for identification such as a barcode. This identification is realized by comparing the image data with the pattern matching using the existing method.

Alternatively, the identification unit 312 realizes this identification, for example, by constructing a learning model by machine learning. In this case, for example, the identification unit 312 has image data of an article that can be any of the inspection objects 4 as a subject, and a label indicating the type of the inspection object 4 that is the subject (that is, a label indicating a correct answer). Generate teacher data in combination with. In this case, teacher data in which an image data in which none of the inspection objects 4 is a subject and a label indicating an incorrect answer may be further generated. Then, the identification unit 312 performs supervised machine learning using this teacher data. In this case, for example, the identification unit 312 performs machine learning by a neural network configured by combining perceptrons. Specifically, the feature amount of the image data included in the teacher data is given to the input layer of the neural network as input data, and weighting is performed for each perceptron so that the output of the output layer of the neural network is the same as the label. Repeat learning while changing.

Then, after the learning model is constructed in this way, the identification unit 312 inputs the feature amount of the image data of the inspection object into the learning model, and uses the output as the identification result. The machine learning method is not always limited, and for example, a method suitable for image analysis such as a convolutional neural network (CNN) may be used.

The identification unit 312 identifies, by image analysis using artificial intelligence or a predetermined algorithm, which type of inspection object 4 the subject 4 is, and identifies the inspection object 4. The result is transmitted to the wearable terminal 2. As described above, in the present embodiment, the inspection target 4 can be identified by image analysis. Therefore, even if a barcode or an IC tag is not attached to the inspection object 4, a wide variety of inspection objects 4 can be identified. That is, the present embodiment can be applied with the current distribution mechanism without the need for modification or processing of the inspection object 4.

The work information management unit 313 manages work information in the same manner as the work information management unit 215 included in the wearable terminal 2. Here, since the details of the work information are described above in the explanation of the work information management unit 215, the description thereof will be omitted again here. Further, the work information management unit 313 is also the same as the work information management unit 215, but communicates with the work information management unit 215 and the server device (not shown) of the logistics company that delivers the inspection object 4. By doing so, work information is acquired and updated.
Then, the work information management unit 313 stores the work information acquired and updated in this way in the work information storage unit 352. That is, the work information storage unit 352 functions as a storage unit for storing work information.

The object information managed by the object information management unit 213 and the object information management unit 311 and the work information managed by the work information management unit 215 and the work information management unit 313 are the wearable terminal 2 and the server as described above. It may be updated by a method other than communication between the device 3 and the server of each business operator. For example, the object information and the work information may be updated by being added or modified based on the input operation by the user U.

[Status judgment process]
The configurations of the state determination device 1, the wearable terminal 2, and the server device 3 in the present embodiment have been described in detail. Next, the processing contents of each processing included in the state determination processing executed by the state determination system S including each of these devices will be described with reference to the flowcharts of FIGS. 11, 12, 13, and 16. FIG. 11 is a flowchart illustrating a flow of preprocessing in the state determination process. FIG. 12 is a flowchart illustrating the flow of the gripping start process in the state determination process. FIG. 13 is a flowchart illustrating a flow of the determination process in the state determination process. FIG. 16 is a flowchart illustrating a flow of the gripping end process in the state determination process. As a premise of the explanation, the object information management unit 213 and the object information management unit 311 manage the object information, and the work information management unit 215 and the work information management unit 313 manage the work information separately. It is assumed that there is.

The state determination process is executed in accordance with the start instruction operation of the state determination process for the wearable terminal 2 from the user U. It should be noted that, for example, the instruction operation in step S1 and step S6, which will be described later, including this instruction operation may be performed based on the instruction operation consisting of voice instructions and gestures by the user U, as described above. It's a street.

First, referring to FIG. 11, in step S1, the object shooting unit 212 takes a picture using the camera 28 based on the shooting instruction operation from the user U, so that the inspection target 4 is the subject of the inspection. Generate object image data.
In step S2, the object photographing unit 212 transmits the generated image data of the inspection object to the server device 3.

In step S3, the identification unit 312 identifies which type of inspection object 4 the inspection object 4 as the subject is by analyzing the image data of the inspection object.
In step S4, the identification unit 312 transmits the identification result (that is, which type of inspection object 4 is) to the wearable terminal 2. In this case, the identification unit 312 reads not only the identification result but also the object information of the inspection object 4 corresponding to the identification result from the object information management unit 351 and this object information is also sent to the wearable terminal 2. Send. However, the identification unit 312 may transmit only the identification result, and the wearable terminal 2 that receives the identification result may read the object information from the object information storage unit 251.

In step S5, the notification unit 214 notifies the user U of the object information corresponding to the identification result. For example, the product name and the image data of the appearance (that is, the packaging container) included in the object information are displayed for notification.

In step S6, the notification unit 214 determines whether or not the identification result by the server device 3 is correct. This determination is made based on whether or not an operation indicating that the identification result is correct is received from the user U who has compared the notification content in step S5 with the actual inspection target object 4. If the operation to the effect that the identification result is correct is accepted, it is determined as Yes in step S6, and the process proceeds to step S7. On the other hand, if the operation to the effect that the identification result is correct is not accepted, it is determined as No in step S6, the process returns to step S1, and the shooting of step S1 is repeated again.

In step S7, the object information management unit 213 transmits the object information corresponding to the identification result to the state determination device 1.
In step S8, the determination standard setting unit 113 sets a predetermined threshold value included in the object information corresponding to the identification result as a threshold value as a reference for determining whether or not the inspection object 4 is damaged. .. Although the details will be described later, in the present embodiment, two threshold values, a first threshold value and a second threshold value, are set as reference threshold values for determination.

In step S9, the determination standard setting unit 113 transmits to the wearable terminal 2 that the setting of the reference threshold value has been completed.
In step S10, the notification unit 214 notifies the user U of the status indicating the state of the state determination device 1 as "ready".

In step S11, the notification unit 214 notifies the user U of the image data of the position to be gripped by the gripping mechanism 65 at the time of inspection included in the object information corresponding to the identification result. In this case, for example, the image data of the position to be gripped may be displayed together with the image data of the packaging container.
However, the present invention is not limited to this, and the image data of the position to be gripped may be superimposed and displayed so as to match the position of the actual packaging container by using the technique of mixed reality. In this case, a text such as "Please start gripping" may be displayed or a voice may be output. By doing so, it becomes possible for the user U to easily recognize the position to be gripped and the timing to start gripping.

With the notification in step S11, the user U starts an operation on the operation mechanism 55 in order to grip the inspection object 4.

Next, referring to FIG. 12, in step S12, the operation control unit 111 operates as a master unit 50 or a slave unit in response to an operation on the operation mechanism 55 of the user U. For 60, the control of the movement to which the force / tactile transmission function is applied is started. This operation control continues until the operation of the user U on the operation mechanism 55 is completed (for example, until step S33 described later).

In step S13, the parameter acquisition unit 112 acquires the control parameters related to the current force-tactile sensation calculated in real time.
In step S14, it is determined whether or not the value indicating the force in the control parameter related to the force-tactile sensation acquired in step S13 is equal to or greater than the first threshold value. This determination is made with the intention of confirming that the inspection object 4 is being gripped by the gripping mechanism 65. Therefore, the first threshold value is set to a value equal to or higher than the value of the force capable of gripping the inspection object 4 by the gripping mechanism 65. The value of the force that can be gripped differs depending on the type of the object to be inspected 4. Therefore, as described above, in the present embodiment, the first threshold value is different depending on the type of the inspection object 4.

The "value indicating the force" in this determination to be compared with the first threshold value is, for example, the slave by moving the first gripping member included in the gripping mechanism 65 by the first actuator included in the slave side actuator 62. The second gripping member included in the gripping mechanism 65 is moved by the second actuator included in the side actuator 62, and the distance between the first gripping member and the second gripping member is shortened (for example, by closing) to grip. In the case of such a configuration, each of the value indicates the force corresponding to the first actuator and the value indicates the force corresponding to the second actuator.

If the value indicating the force is equal to or greater than the first threshold value, Yes is determined in step S14, and the process proceeds to step S16. On the other hand, if the value indicating the force is less than the first threshold value, it is determined as No in step S14, and the process proceeds to step S15.

As described above, the determination unit 114 determines whether or not the value indicating the force is equal to or higher than the first threshold value, and does not determine the upper limit of the value indicating the force. However, when the threshold value is equal to or higher than the first threshold value and the grip is performed with an excessive force, the determination conditions are not constant and it becomes difficult to make a determination with stable accuracy. Further, even if the inspection object 4 is originally not damaged, it may be damaged due to this gripping. Therefore, in order to prevent the determination accuracy from becoming unstable and damage from occurring, the force in gripping should not be excessive (that is, the force in gripping should be a certain strength or more). It is advisable to carry out further suppression processing (so that it does not become).

For that purpose, for example, the above-mentioned (scaling function) is applied in the operation control by the operation control unit 111. When this scaling function is applied, the scale of the output force can be scaled up or down with respect to the reference control. Then, when the value indicating the force becomes a predetermined value or more by this scaling function, the motion control unit 111 reduces the strength (force) of the movement of the master device and reproduces it in the slave device. That is, even if the motion strength (force) of the master device becomes excessive, the motion control unit 111 reduces the motion so that the slave device does not exceed the predetermined motion strength (force). Control. As a result, even if the master device is operated with an excessive force, the gripping by the slave side unit 60 and the gripping mechanism 65 is suppressed so as not to be an excessive force. Therefore, it is possible to prevent the determination accuracy from becoming unstable and the inspection object 4, which is not originally damaged, from being damaged due to gripping.

In step S15, the determination unit 114 resets the value of the counter used in steps S13 to S17. Then, the process returns to step S13 and is repeated. This is a process performed with the intention of resetting the value of the counter because the inspection object 4 is not yet gripped by the gripping mechanism 65.

In step S16, the determination unit 114 counts up the value of the counter by one.
In step S17, the determination unit 114 determines whether or not the value of the counter has reached the specified count number. When it is reached, it is determined as Yes in step S17, and the process proceeds to step S18. On the other hand, if it has not been reached, it is determined as No in step S17, the process returns to step S13, and the process is repeated. This is a process performed with the intention of confirming that the gripping is properly performed and the next process can be performed thereafter because the state in which the force is equal to or higher than the first threshold value continues for a predetermined time.

In step S18, the determination unit 114 stores the control parameters related to force and tactile sensation acquired by repeating step S13 in the parameter storage unit 151. This stored control parameter is then used for reference by the user U in a situation where the value of the first threshold value corresponding to the type of the inspection object 4 is corrected.

In step S19, the determination unit 114 transmits to the wearable terminal 2 that the determination of whether or not the inspection object 4 is damaged is started.
In step S20, the notification unit 214 notifies the user U of the status indicating the state of the state determination device 1 as "determining".

Next, referring to FIG. 13, in step S21, the parameter acquisition unit 112 acquires the control parameters related to the current force-tactile sensation calculated in real time.
In step S22, the determination unit 114 determines whether or not the value indicating the force in the control parameter related to the force tactile sensation acquired in step S21 maintains the first threshold value or more. This determination is a process performed with the intention of confirming that the state in which the inspection object 4 is gripped by the gripping mechanism 65 is properly continued.

When the value indicating the force maintains the first threshold value or more, it is determined as Yes in step S22, and the process proceeds to step S25. On the other hand, when the value indicating the force does not maintain the first threshold value or more (that is, when the value indicating the force is less than the first threshold value), it is determined as No in step S22, and the process proceeds to step S24.

In step S23, the determination unit 114 transmits to the wearable terminal 2 that the failure to grip the inspection object 4 has been detected.
In step S24, the notification unit 214 notifies the user U of the start that indicates the state of the state determination device 1 as an "error and retry". Then, the process returns to step S10 and is repeated.

In step S25, the determination unit 114 determines whether or not the value indicating the position in the control parameter related to the force-tactile sensation acquired in step S21 is less than the second threshold value. This determination is a process performed with the intention of confirming that the inspection object 4 is not damaged by the gripping mechanism 65.

In this respect, if the inspection object 4 is not damaged, leakage of gas such as carbon dioxide or nitrogen gas (so-called air leakage) or outflow of food or the like, which are the contents, do not occur. Therefore, even if the gripping is continued, the packaging container which is the inspection object 4 should not be crushed more than a predetermined standard. On the other hand, if the inspection object 4 is damaged, gas leakage (so-called air leakage) such as carbon dioxide and nitrogen gas, which are the contents, and outflow of food and the like occur. Therefore, if the gripping is continued, the packaging container which is the inspection object 4 is crushed more than a predetermined standard.

Therefore, the second threshold value is a value at which the value indicating the position does not become less than that when the inspection object 4 is gripped by the gripping mechanism 65 if the inspection object 4 is not damaged (that is, the above-mentioned value). It is set to a value that indicates a position that should not move unless it is crushed above a predetermined standard. For example, when the gripping mechanism 65 opens and closes to grip the inspection object 4, if the inspection object 4 is not damaged, a value indicating a position indicating a position where the inspection object 4 should not be closed is set. It is set to the second threshold. The above-mentioned predetermined criteria differ depending on the type of the inspection object 4. Therefore, as described above, in the present embodiment, the second threshold value is different depending on the type of the inspection object 4.

The "value indicating the position" in this determination to be compared with the second threshold value is, for example, a relative value based on the position of the actuator before the start of gripping. For example, in the case of a mechanism for gripping by opening and closing the gripping member included in the gripping mechanism 65, the relative value is based on the position of the actuator in the open state before the start of gripping.
In this case, in order to further improve the accuracy of the determination, it is desirable that the reference state before the start of gripping is the same every time. Therefore, for example, in the operation control by the operation control unit 111, when it is detected that the value indicating the force is less than a predetermined value and the grip is not performed, the slave side actuator 62 is controlled to grip the actuator. Control is performed so that the gripping member included in the mechanism 65 moves to the same position each time. For example, in the case of a mechanism for gripping by opening and closing the gripping member included in the gripping mechanism 65, the slave side actuator 62 is controlled so as to open with the same width each time. As a result, the reference state before the start of gripping becomes the same every time, so that the accuracy of the determination can be further improved.

As another method, the "value indicating the position" in this determination means, for example, that the first gripping member included in the gripping mechanism 65 is moved by the first actuator included in the slave side actuator 62, and the slave side actuator is used. The second actuator included in 62 moves the second gripping member included in the gripping mechanism 65, and the distance between the first gripping member and the second gripping member is shortened (for example, by closing) to perform gripping. In the case of such a configuration, the value indicating the position corresponding to the first actuator and the value indicating the position corresponding to the second actuator may be relative values with respect to the other.

If the value indicating the position is less than the second threshold value, it is determined as Yes in step S25, and the process proceeds to step S28. This is because the position is less than the second threshold value and the inspection object 4 remains crushed by a predetermined standard or more, so that the inspection object 4 is determined to be damaged. It is a process to be done. On the other hand, if the value indicating the position is less than the second threshold value, it is determined as No in step S25, and the process proceeds to step S26.

In step S26, the determination unit 114 counts up the value of the counter used in steps S21 to S27 by one.
In step S27, the determination unit 114 determines whether or not the value of the counter has reached the specified count number. When it is reached, it is determined as Yes in step S27, and the process proceeds to step S28. On the other hand, if it has not been reached, it is determined as No in step S27, the process returns to step S21, and the process is repeated. This is because the state in which the position does not fall below the second threshold value continues for a predetermined time, and the inspection object 4 remains in a state in which the inspection object 4 is not crushed by a predetermined reference or more even if the gripping is performed. This process is performed with the intention of confirming that the product is not damaged.

As described above, the determination unit 114 damages the inspection object 4 based on whether the value indicating the force maintains the first threshold value and the value indicating the position is not less than the second threshold value. Make a judgment about. That is, the determination is made based on the change in position (that is, the amount of movement) while the force in gripping is constant. In this way, the determination unit 114 appropriately specifies the impedance in gripping based on the control parameters related to the force and tactile sensation, and objectively determines a predetermined reference (here, a reference of the amount of movement when the force is constant and in contact). By making a judgment based on various indicators, it is possible to realize a judgment with stable accuracy.

In step S28, the determination unit 114 stores the control parameters related to force and tactile sensation acquired by repeating step S21 in the parameter storage unit 151. This stored control parameter is then used for reference by the user U in a situation where the value of the second threshold value corresponding to the type of the inspection object 4 is corrected.

In step S29, the determination unit 114 transmits the determination result of whether or not the inspection object 4 is damaged to the wearable terminal 2. Here, in step S29, which is performed after the determination of Yes in step S25, the transmitted determination result is "the inspection object 4 is damaged." On the other hand, in step S29 performed after the determination of Yes in step S27, the transmitted determination result is "the inspection object 4 is not damaged."

In step S30, the notification unit 214 notifies the user U of the determination result. For example, a text indicating either "the inspection object 4 is damaged" or "the inspection object 4 is not damaged", which is the determination result, is displayed or a voice is output. Or something.
In step S31, the notification unit 214 notifies the user U of the work information. For example, this work information is based on the determination result, and the content is such that the user U specifically instructs the work to be performed next.

Specific examples of the notifications in steps S30 and S31 will be described with reference to FIGS. 14 and 15. FIG. 14 is a schematic diagram showing a notification example when it is determined that the inspection object 4 is not damaged. FIG. 15 is a schematic diagram showing a notification example when it is determined that the inspection object 4 is damaged.

FIG. 14 shows a transmissive display included in the output unit 27. Since this display is a transmissive type, the user U has a state determination device 1 operated by himself / herself via the display, an inspection object 4 held by the state determination device 1, a container as a storage destination, and a container. You can see the real thing. In addition, in FIGS. 14 and 15, the real thing which is visually recognized through these displays is represented by a broken line. In addition, the information notified by the notification unit 214 is displayed on the display. Specifically, in the area AR1 on the display, the text "OK" corresponding to the determination result that the inspection object 4 is not damaged and the value indicating the position ("measured value" in the figure) are the first. 2 Text indicating that the value is less than the threshold value (“threshold value” in the figure) is displayed. Further, in the area AR2 on the display, the number of each product to be stored in the work and the number of products lacking in the number to be stored are displayed. This display can be generated based on the work information or the like stored in the work information storage unit 252. Further, in the area AR3 on the display, a text indicating the storage destination of the inspection object 4 "stored in the container ABC" is displayed as a content for specifically instructing the work to be performed next by the user U.

FIG. 15 also shows a transmissive display included in the output unit 27. The user U can visually recognize the actual state determination device 1 operated by the user U via the display, the inspection object 4 held by the state determination device 1, and the container to be discarded. In addition, in the area AR4 on the display, the text "NG" corresponding to the determination result that the inspection object 4 is damaged and the value indicating the position ("measured value" in the figure) are second. Text indicating that the threshold (“threshold” in the figure) has been exceeded is displayed. Further, in the area AR5 on the display, a text indicating the disposal destination of the inspection object 4 "discard in the disposal container" is displayed as the content for specifically instructing the work to be performed next by the user U.

By referring to the displays as illustrated in FIGS. 14 and 15, the user U can easily grasp various information such as the determination result, the progress of the work, and the work to be performed next by the user U. Can be done. That is, in the present embodiment, the technique of transmitting force and tactile sensation and the technique of mixed reality can be combined to assist (that is, support) the work and judgment in the inspection and the like of the user U. Further, since the inspection work can be performed in parallel with the work such as sorting and picking, the work process can be compressed.

Next, referring to FIG. 16, in step S32, the parameter acquisition unit 112 acquires the control parameters related to the current force-tactile sensation calculated in real time.
In step S33, the determination unit 114 determines whether or not the value indicating the force in the control parameter related to the force tactile sensation acquired in step S32 is less than the first threshold value. In this determination, the user U completes the work with reference to the notification as shown in FIGS. 15 and 16 (for example, the inspection object 4 is stored in the container), and the inspection object 4 is gripped by the gripping mechanism 65. It is done with the intention of confirming that the current state has ended.

If the value indicating the force is less than the first threshold value, Yes is determined in step S33, and the process proceeds to step S34. On the other hand, when the value indicating the force is equal to or higher than the first threshold value (that is, when the grip is still continued), No is determined in step S33, the process returns to step S32, and the determination is repeated.

In step S34, the determination unit 114 stores the control parameters related to force and tactile sensation acquired by repeating step S32 in the parameter storage unit 151. This stored control parameter is then used for reference by the user U in a situation where the value of the first threshold value corresponding to the type of the inspection object 4 is corrected.

In step S35, the determination unit 114 transmits to the wearable terminal 2 that a series of state determination processes have been completed.
In step S30, the notification unit 214 notifies the user U that a series of state determination processes have been completed.
In step S37, the notification unit 214 initializes the start test indicating the state of the state determination device 1.

In step S38, the work information management unit 215 updates the work information stored in the work information storage unit 252, reflecting the current work content. For example, if the inspection object 4 is not damaged and is stored in a container, the current number of stored objects in this container is increased. Alternatively, if the inspection object 4 is damaged and discarded, the number of discarded objects is increased. In addition, information such as the identification information of the user U who has performed these operations and the work end time is also added.
In step S39, the work information management unit 215 transmits the updated work information to the server device 3.

In step S40, the work information management unit 313 updates the work information stored in the work information storage unit 352 based on the received updated work information. This ends this process. In this way, by performing the processes of steps S38 to S40, not only the work and judgment in the inspection and the like of the user U are assisted (that is, support), but also the current work information can be easily updated and finally. It is possible to easily create various work histories.

According to the state determination process described above, when inspecting for damage, it is possible to realize an inspection with stable accuracy based on an objective index.
Further, according to the state determination process, the inspection work can be performed in parallel with the work such as sorting and picking, that is, the work process can be compressed.

<Second Embodiment>
The first embodiment of the present invention has been described above. Subsequently, the second embodiment of the present invention will be described below. Hereinafter, the configuration and processing contents peculiar to the second embodiment, which are different from the first embodiment, will be described in particular detail. On the other hand, the contents overlapping between the first embodiment and the second embodiment will be omitted again as appropriate.

[Summary of differences from the first embodiment]
First, the differences between the first embodiment and the second embodiment will be briefly described. In the first embodiment, when the state determination process is performed, in step S25, the damage of the inspection object 4 is determined by comparing the control parameter relating to the force-tactile sensation with the second threshold value.

On the other hand, in the second embodiment, the "machine learning process" is performed in advance before the state determination process is performed. Here, in the machine learning process, the control parameters related to the force-tactile sensation and the inspection object 4 are subjected to machine learning using the set of the control parameters related to the force-tactile sensation and the determination result regarding the damage of the inspection object 4 as the teacher data. It is a series of processes to build a machine learning model in which the relationship with the judgment result regarding the damage of is machine-learned.
Then, in the second embodiment, the state determination process is performed after the machine learning process. In this state determination process, the control parameters related to the force-tactile sensation used in the motion control by the motion control unit 111 when grasping the inspection object 4 to be determined are input to the machine learning model constructed in the machine learning process. Make a judgment about damage.
As a result, in the second embodiment, the judgment is made based on an objective index called a machine learning model in which the relationship between the control parameter and the judgment result regarding the damage of the inspection object 4 is machine-learned, which is more stable. Inspection with accuracy can be realized.

[System configuration], [Operation control for controlled device] and [Status determination device configuration]
The overall configuration of the state determination system S according to the present embodiment is the same as the overall configuration of the first embodiment described with reference to FIG. Further, it is assumed that the inspection object 4 is a packaging container for packaging food (for example, sweets such as potato chips) as the content, and a value indicating the position as a control parameter related to force and tactile sensation is assumed. The points are the same as those in the first embodiment.
Further, the basic principle of operation control for the controlled object device (here, the master side unit 50, the operation mechanism 55, the slave side unit 60, and the gripping mechanism 65 included in the state determination device 1) in the present embodiment is also described. It is the same as the basic principle of the operation control of the first embodiment described with reference to FIGS. 2 to 6.
Further, the basic configuration of the state determination device 1 in the present embodiment is the same as the basic configuration of the state determination device 1 of the first embodiment described with reference to FIG. 7, except for the control unit 10. ..
Therefore, duplicate description of these points will be omitted.

In the present embodiment, the control unit 10 (hereinafter, the control unit 10 in the present embodiment is referred to as a "control unit 10a") operates as a master device of the master side unit 50 or as a slave device of the slave side unit 60. By controlling the operation of the above, the function as a gripping device is realized, and further, by cooperating with the wearable terminal 2 and the server device 3, the above-mentioned machine learning process and state determination process are performed.

FIG. 17 is a block diagram showing an example of hardware and functional blocks of the control unit 10a for realizing the machine learning process and the state determination process. The control unit 10a of the present embodiment further includes a sensor 19 as hardware as a difference from the control unit 10 of the first embodiment.

The sensor 19 is composed of a group of sensors that measure the ambient temperature (that is, air temperature), humidity, and atmospheric pressure of the state determination device 1 as information indicating the environment around the state determination device 1 provided with the control unit 10a. The measurement by the sensor 19 is performed, for example, in the machine learning process or the state determination process, while the control unit 10a makes the state determination device 1 function as a gripping device and grips the inspection object 4. The measurement result of the sensor 19 is output to the processor 11 as appropriate.

When machine learning processing and state determination processing are realized in such a hardware configuration, as shown in FIG. 17, the operation control unit 111, the parameter acquisition unit 112, the correct answer information acquisition unit 115, and the processor 11 are used. The environment information acquisition unit 116 and the processing unit 117 function. As described above, in the control unit 10a of the present embodiment, the correct answer information acquisition unit 115, the environment information acquisition unit 116, and the processing unit 117 function as differences from the control unit 10 of the first embodiment. Therefore, the determination standard setting unit 113 and the determination unit 114 do not function.
Further, in the case of realizing machine learning processing and state determination processing in such a hardware configuration, as shown in FIG. 17, one area of the storage unit 15 includes a parameter storage unit 151, a correct answer information storage unit 153, and the like. The environmental information storage unit 154 and the like are set. As described above, in the control unit 10a of the present embodiment, the correct answer information storage unit 153 and the environmental information storage unit 154 are set as differences from the control unit 10 of the first embodiment, while the determination criteria are set. The storage unit 152 is not set.

The correct answer information acquisition unit 115 acquires "correct answer information" used as a part of teacher data in the machine learning process. In the machine learning process, the state determination device 1 is made to function as a gripping device for the purpose of creating teacher data for performing machine learning, and the inspection object 4 is gripped. The correct answer information is a determination result as to whether or not the gripped inspection object 4 is damaged, and is used as information indicating the correct answer (that is, a label indicating the correct answer) in machine learning.

The correct answer information is, for example, whether the inspection object 4 is damaged, which is determined by the expert user U based on the feedback of the reaction force against the operation mechanism 55 when the gripping mechanism 65 grips the inspection object 4. It is a judgment result of whether or not. As a result, for example, the user U, who is an expert, can only inspect the inspection object 4 as part of the normal work, and can further collect teacher data. Alternatively, the correct answer information is, for example, a determination result by the user U who knows in advance a determination result as to whether or not the inspection object 4 is damaged. Since the correct answer information is used as information indicating the correct answer in machine learning, only the correct judgment result by the user U is acquired as the correct answer information in order to improve the accuracy of machine learning.

Such correct answer information is input, for example, by the wearable terminal 2 receiving an instruction consisting of voice or an instruction operation consisting of gestures from the user U, and is acquired by the correct answer information acquisition unit 115 by communication via the network N. To.
Then, the correct answer information acquisition unit 115 stores the correct answer information acquired in this way in the correct answer information storage unit 153 as a part of the teacher data. That is, the correct answer information storage unit 153 functions as a storage unit for storing correct answer information.

The environmental information acquisition unit 116 acquires "environmental information" to be used as a part of teacher data in machine learning processing or as a part of input data in state determination processing.
As described above, in the machine learning process, the state determination device 1 is made to function as a gripping device for the purpose of creating teacher data for performing machine learning, and the inspection object 4 is gripped. Further, also in the state determination process, the state determination device 1 is made to function as a gripping device for the purpose of determining the damage of the inspection object 4, and the inspection object 4 is gripped.
The environmental information is the temperature, humidity, and atmospheric pressure around the state determination device 1 measured by the sensor 19 when these are gripped.

The reason for using the environmental information in this embodiment will be described. It is generally known that changes in barometric pressure and changes in temperature are related. For example, when the atmospheric pressure drops, the temperature also drops. Then, when these atmospheric pressures and temperatures rise or fall, the gas inside the inspection object 4, which is a packaging container, contracts or expands. For example, when the inspection target 4 is transported to a highland, the gas in the inspection target 4 expands as the ambient air pressure and temperature decrease, and as a result, the inspection target 4 itself also expands. Since such an event occurs, the atmospheric pressure and the temperature affect the change in the value of the control parameter related to the force and tactile sensation when grasping.

Regarding humidity, for example, the contents rot due to humidity and the volume increases, and the water vapor in the packaging container increases in conjunction with changes in temperature and atmospheric pressure, causing the container to expand. Can also affect changes in the values of control parameters related to force and tactile sensation when gripping.
Therefore, in the present embodiment, environmental information such as the ambient temperature, humidity, and atmospheric pressure of the state determination device 1 is taught in consideration of the influence on the change of the value of the control parameter related to the force and tactile sensation caused by such a difference in environment. Perform machine learning as part of the data. As a result, a machine learning model capable of making a judgment with higher accuracy can be constructed, and a judgment regarding damage can be made.

Such environmental information is measured by, for example, the sensor 19 and acquired by the environmental information acquisition unit 116.
Then, the environmental information acquisition unit 116 stores the environmental information acquired in this way in the environmental information storage unit 154 as a part of the teacher data or a part of the input data. That is, the environmental information storage unit 154 functions as a storage unit for storing environmental information.

In addition, in parallel with the acquisition of the correct answer information and the acquisition of the environmental information by the correct answer information acquisition unit 115 and the environmental information acquisition unit 116, the parameter acquisition unit 112 also acquires the control parameters related to the force-tactile sensation.
Then, the parameter acquisition unit 112 stores the acquired control parameter related to force and tactile sensation in the parameter storage unit 151 in order to use it as a part of the teacher data or as a part of the input data in the state determination process. That is, the parameter storage unit 151 functions as a storage unit for storing control parameters related to force and tactile sensation.

The processing unit 117 executes various processes such as sending and receiving various information with other devices in the machine learning process and the state determination process. For example, in machine learning processing, when the processing unit 117 grips a certain inspection object 4 for creating teacher data, the control parameter related to force and tactile sensation acquired by the parameter acquisition unit 112, the correct answer information acquisition unit 115. The correct answer information (that is, the label indicating the correct answer) acquired by the above and the environmental information acquired by the environmental information acquisition unit 116 are read out from the corresponding storage units to form a set. Then, the processing unit 117 transmits this set of information as teacher data to the server device 3.

In addition, for example, in the state determination process, the processing unit 117 has a control parameter related to force and tactile sensation acquired by the parameter acquisition unit 112 when the inspection object 4 to be determined is grasped, and an environmental information acquisition unit 116. Each of the environmental information acquired by is read from the corresponding storage unit and made into a set. Then, the processing unit 117 transmits this set of information as input data to the server device 3.
The machine learning method using the teacher data and the determination method using the input data by the server device 3 will be described later with reference to the block diagram of FIG.

[Wearable device configuration]
The hardware and functional blocks of the wearable terminal 2 in the present embodiment are the same as the hardware and functional blocks of the wearable terminal 2 of the first embodiment described with reference to FIG. Therefore, a duplicate description will be omitted on this point. A part of the processing of the wearable terminal 2 is different from that of the first embodiment, but this point will be described later with reference to the flowcharts of FIGS. 19 to 22.

[Server device configuration]
Next, the configuration of the server device 3 (hereinafter, the server device 3 in the present embodiment is referred to as “server device 3a”) will be described with reference to FIG. FIG. 18 is a block diagram showing an example of hardware and functional blocks of the server device 3a. As shown in FIG. 18, the server device 3a includes the same hardware as the server device 3 of the first embodiment.

When machine learning and state determination processing are realized in such a hardware configuration, as shown in FIG. 18, the object information management unit 311, the identification unit 312, the work information management unit 313, and the processor 31 are used. The machine learning unit 314 and the determination unit 315 function. As described above, in the server device 3a of the present embodiment, the machine learning unit 314 and the determination unit 315 function as differences from the server device 3 of the first embodiment, and in this case, it is shown in FIG. As described above, in one area of the storage unit 35, an object information management unit 351, a work information storage unit 352, a teacher data storage unit 353, and a learning model storage unit 354 are set. As described above, in the server device 3a of the present embodiment, the teacher data storage unit 353 and the learning model storage unit 354 are set as differences from the server device 3 of the first embodiment.

The machine learning unit 314 constructs a machine learning model for determining damage to the inspection object 4 by performing machine learning in the machine learning process.
Therefore, the machine learning unit 314 first acquires teacher data from the state determination device 1. As described above, this teacher data includes control parameters related to force and tactile sensation, correct answer information (that is, a label indicating the correct answer), and correct answer information acquired when the inspection object 4 is grasped to create the teacher data. It is a set of environmental information. In addition, other information may be used as teacher data. For example, a set of control parameters related to force and tactile sensation, a label indicating an incorrect answer, and environmental information acquired when grasping an object other than the inspection object 4 may be used as teacher data.

Then, the machine learning unit 314 uses the teacher data to perform supervised machine learning. In this case, for example, the machine learning unit 314 performs machine learning by a neural network configured by combining perceptrons. Specifically, data features such as force-tactile control parameters and environmental information included in the teacher data are given as input data to the input layer of the neural network, and the output of the output layer of the neural network is correct information (that is, that is). The learning is repeated while changing the weighting for each perceptron so as to be the same as the label indicating the correct answer).
Then, the machine learning unit 314 ends learning when a predetermined condition (for example, the output becomes a correct answer with a higher accuracy than a predetermined value or the learning is repeated for a predetermined time) is satisfied, and at that time. The machine learning model constructed in is stored in the learning model storage unit 354.
That is, the learning model storage unit 354 functions as a storage unit for storing the machine learning model. It should be noted that the construction includes not only creating a new machine learning model but also re-learning the machine learning model stored in the learning model storage unit 354 using new teacher data. ..

The machine learning method is not always limited, and other methods suitable for classification problems such as the nearest neighbor method, decision tree, random forest, and support vector machine may be used. Further, the machine learning unit 314 may construct only one machine learning model, but constructs a plurality of machine learning models according to each of the types of inspection objects 4 having different materials and sizes to be gripped. You may. That is, the machine learning model may be used properly according to the type of the inspection object 4.

In the state determination process, the determination unit 315 makes a determination regarding the damage of the inspection object 4 by using the machine learning model.
Therefore, the determination unit 315 first acquires input data for determination from the state determination device 1. As described above, this input data is a set of control parameters related to force and tactile sensation and environmental information acquired when the inspection object 4 to be determined is grasped.
Then, the determination unit 315 makes a determination regarding the damage of the inspection object 4 by using this input data and the machine learning model stored in the learning model storage unit 354. If the machine learning model is used properly according to the type of the inspection object 4, the determination unit 315 naturally performs machine learning according to the type of the inspection object 4 to be determined this time. Judgment is made using the model.

For example, the determination unit 315 inputs the input data to the machine learning model, and uses the output as the determination result of the determination regarding the damage of the inspection object 4. In this case, the determination result is information indicating whether or not the inspection object 4 is damaged.
Further, when the determination unit 315 determines the damage of the inspection object 4 by the state determination process using the machine learning model in this way, the determination unit 315 transmits the determination result to the wearable terminal 2. Then, this determination result is notified to the user from the wearable terminal 2.

As described above, in the present embodiment, the inspection object 4 is objectively based on the control parameters related to the force and tactile sensation when the inspection object 4 is gripped by the state determination device 1 and the environmental information such as temperature, humidity and atmospheric pressure. Make a judgment about the damage of. Therefore, in the present embodiment, as in the first embodiment, the work can be visualized by converting the sensation (here, force and tactile sensation) in the work into data. As a result, the inspection accuracy does not depend on the personal feeling (that is, the subjectivity of the operator), and the inspection can be performed with stable accuracy.
That is, according to the present embodiment, as in the first embodiment, when inspecting for damage, it is possible to realize an inspection with stable accuracy based on an objective index.

[Machine learning process]
The configurations of the state determination device 1 (that is, the state determination device 1 including the control unit 10a), the wearable terminal 2, and the server device 3a in the present embodiment have been described in detail. Next, the processing contents of each processing included in the machine learning processing executed by the state determination system S including each of these devices will be described with reference to the flowcharts of FIGS. 19 and 20. FIG. 19 is a flowchart illustrating the flow of preprocessing in the machine learning process. FIG. 20 is a flowchart illustrating the flow of the construction process in the machine learning process.

After this pretreatment, the gripping start processing is performed, and then the construction processing is further performed. This gripping start processing is the same as the gripping start processing of the first embodiment described with reference to FIG. Therefore, a duplicate description will be omitted on this point. Further, in the flowchart of the present embodiment (that is, FIGS. 19 to 22), the steps having the same reference numerals as those of the flowchart of the first embodiment (that is, FIGS. 11, 12, 13, and 16) are attached. The processing content is the same as the processing content in the first embodiment. Therefore, the detailed description of the processing contents of these steps will be omitted as appropriate.
Further, as a premise of the explanation, it is assumed that the object information management unit 213 and the object information management unit 311 manage the object information separately.

The machine learning process is executed with the start operation of the machine learning process for the wearable terminal 2 from the user U. It should be noted that, for example, the instruction operation in step S1 and step S6, which will be described later, including this instruction operation may be performed based on the instruction operation consisting of voice instructions and gestures by the user U, as described above. It's a street.

First, referring to FIG. 19, in step S1, the object shooting unit 212 takes a picture using the camera 28 based on the shooting instruction operation from the user U, so that the inspection target 4 is the subject of the inspection. Generate object image data.
In step S2, the object photographing unit 212 transmits the generated image data of the inspection object to the server device 3a.

In step S3, the identification unit 312 identifies which type of inspection object 4 the inspection object 4 as the subject is by analyzing the image data of the inspection object.
In step S4, the identification unit 312 transmits the identification result (that is, which type of inspection object 4 is) to the wearable terminal 2.

In step S5, the notification unit 214 notifies the user U of the object information corresponding to the identification result. For example, the product name and the image data of the appearance (that is, the packaging container) included in the object information are displayed for notification.

In step S6, the notification unit 214 determines whether or not the identification result by the server device 3a is correct. If the operation to the effect that the identification result is correct is accepted, it is determined as Yes in step S6, and the process proceeds to step S7. On the other hand, if the operation to the effect that the identification result is correct is not accepted, it is determined as No in step S6, the process returns to step S1, and the shooting of step S1 is repeated again.

In step S7, the object information management unit 213 transmits the object information corresponding to the identification result to the state determination device 1.
In step S51, the processing unit 117 is set to a recording mode for recording various data described later in order to acquire teacher data.

In step S52, the processing unit 117 transmits to the wearable terminal 2 that the recording mode has been set.
In step S10, the notification unit 214 notifies the user U of the status indicating the state of the state determination device 1 as "ready".

In step S11, the notification unit 214 notifies the user U of the image data of the position to be gripped by the gripping mechanism 65 at the time of inspection included in the object information corresponding to the identification result. With the notification in step S11, the user U starts an operation on the operation mechanism 55 in order to grip the inspection object 4.

After that, as described above, the gripping start processing similar to the gripping start processing of the first embodiment described with reference to FIG. 12 is performed. Then, when the gripping start process is completed, the construction process is started. Regarding the step in which the determination unit 114 is the main processing agent in the gripping start processing of the first embodiment, the processing unit 117 is the main processing agent in the gripping start processing of the present embodiment.

Next, referring to FIG. 19, in step S21, the parameter acquisition unit 112 acquires the control parameters related to the current force-tactile sensation calculated in real time.

In step S22, the processing unit 117 determines whether or not the value indicating the force in the control parameter related to the force-tactile sensation acquired in step S21 maintains the first threshold value or more. This determination is a process performed with the intention of confirming that the state in which the inspection object 4 is gripped by the gripping mechanism 65 is properly continued.

When the value indicating the force maintains the first threshold value or more, it is determined as Yes in step S22, and the process proceeds to step S25. On the other hand, when the value indicating the force does not maintain the first threshold value or more (that is, when the value indicating the force is less than the first threshold value), it is determined as No in step S22, and the process proceeds to step S24.

In step S23, the processing unit 117 transmits to the wearable terminal 2 that the failure to grip the inspection object 4 has been detected.

In step S26, the processing unit 117 counts up the value of the counter used in steps S21 to S27 by one. In the construction process, since it is not necessary to perform the determination by the state determination device 1, the steps S24 and S25 in the first embodiment are not performed.
In step S27, the processing unit 117 determines whether or not the value of the counter has reached the specified count number. When it is reached, it is determined as Yes in step S27, and the process proceeds to step S28. On the other hand, if it has not been reached, it is determined as No in step S27, the process returns to step S21, and the process is repeated. This is a process performed with the intention of acquiring as many control parameters as necessary as teacher data regarding force and tactile sensation.

In step S28, the processing unit 117 stores the control parameters related to force and tactile sensation acquired by repeating step S21 in the parameter storage unit 151. This stored force-tactile control parameter is then utilized by machine learning as part of the teacher data for building the machine learning model.

In step S61, the processing unit 117 provides environmental information on the ambient temperature, humidity, and atmospheric pressure of the state determination device 1 measured by the sensor 19 when gripping (for example, when the acquisition of control parameters related to force and tactile sensation is completed). And store it in the environmental information storage unit 154. This stored environmental information is then used by machine learning as part of the teacher data for building a machine learning model.

In step S62, the processing unit 117 acquires control parameters and environmental information related to force and tactile sensation, and transmits to the wearable terminal 2 that the measurement is completed.

In step S63, the notification unit 214 notifies the user U of the data acquisition result. For example, a value indicating a position included in a control parameter related to force and tactile sensation, and a value of the ambient temperature, humidity, and atmospheric pressure of the state determination device 1 included in the environmental information are displayed as text or output by voice. The user U who refers to this includes the contents of the various notified information, feedback of the reaction force against the operation mechanism 55 when the gripping mechanism 65 grips the inspection object 4, and the inspection that the user has grasped in advance. Based on the information on whether or not the object 4 is damaged, it is determined whether or not the inspection object 4 is damaged.

In step S64, the processing unit 117 receives the determination result of whether or not the inspection object 4 is damaged from the user U who has referred to the notification in step S63 as correct answer information. The reception of the correct answer information is realized, for example, by the processing unit 117 receiving an instruction operation consisting of a voice or a gesture from the user U.
In step S65, the processing unit 117 transmits the correct answer information received in step S64 to the state determination device 1.

In step S66, the processing unit 117 stores the correct answer information received from the wearable terminal 2 in the correct answer information storage unit 153, and the correct answer information (that is, a label indicating the correct answer) and the control parameters related to the force and tactile sensation stored in step S28. , And the environment information stored in step S61 as a set is transmitted to the server device 3a as teacher data.

In step S67, the machine learning unit 314 stores the teacher data received from the state determination device 1 in the teacher data storage unit 353.
In step S68, the machine learning unit 314 uses the teacher data stored in step S67 to satisfy a predetermined condition (for example, the output becomes a correct answer with a higher accuracy than a predetermined value or the learning is repeated for a predetermined time). By performing machine learning until it is satisfied, a machine learning model for determining the damage of the inspection object 4 is constructed.
In step S69, the machine learning unit 314 stores the machine learning model constructed in step S68 in the learning model storage unit 354.

In this flowchart, it is assumed that machine learning is performed each time teacher data is acquired, but the present invention is not limited to this. For example, machine learning may be performed after a plurality of teacher data are acquired and accumulated to some extent. Alternatively, instead of constructing a machine learning model in the machine learning process, a machine learning model may be constructed each time a determination is made in the state determination process.
Further, when the machine learning model is used properly according to the type of the inspection object 4, the identification result in step S3 and the machine learning model constructed in step S68 are linked to the learning model storage unit 354. Try to remember. This makes it possible to specify a machine learning model according to the type of the inspection object 4 in the state determination process.

In step S70, the processing unit 117 transmits to the wearable terminal 2 that the storage of the teacher data is completed.
In step S71, the notification unit 214 notifies the user U that the storage of the teacher data is completed.

In step S72, the notification unit 214 notifies the user U to select whether or not to continuously store the state of another inspection object 4. When the user U who responds to this notification receives the reply "remember the state of another inspection object 4", it is determined as Yes in step S72, and the process returns to step S1 in FIG. 19 and is repeated. On the other hand, when the reply "The state of another inspection object 4 is not stored" is received from the user U, it is determined as No in step S72, and the machine learning process ends. It should be noted that the reception of these answers is realized, for example, by the processing unit 117 receiving an instruction operation consisting of a voice or a gesture from the user U.

[Status judgment process]
Next, the processing contents of each process included in the state determination process executed by the state determination system S including the state determination device 1, the wearable terminal 2, and the server device 3a will be described with reference to the flowcharts of FIGS. 21 and 22. do. FIG. 21 is a flowchart illustrating a flow of preprocessing in the state determination process. FIG. 22 is a flowchart illustrating a flow of the determination process in the state determination process.

Similar to the state determination process of the first embodiment, after this preprocessing, the gripping start process is performed, then the construction process is further performed, and finally the gripping end process is performed. This gripping start processing is the same as the gripping start processing of the first embodiment described with reference to FIG. Further, this gripping end process is the same as the gripping end process of the first embodiment described with reference to FIG. Therefore, duplicate description of these points will be omitted. Further, as described above, in the flowchart of the present embodiment (that is, FIGS. 19 to 22), the same reference numerals as those of the flowchart of the first embodiment (that is, FIGS. 11, 12, 13, 13 and 16) are assigned. The processing content of the step is the same as the processing content in the first embodiment. Therefore, the detailed description of the processing contents of these steps will be omitted as appropriate.
Further, as a premise of the explanation, it is assumed that the object information management unit 213 and the object information management unit 311 manage the object information separately.

First, referring to FIG. 21, in step S1, the object shooting unit 212 takes a picture using the camera 28 based on the shooting instruction operation from the user U, so that the inspection target 4 is the subject of the inspection. Generate object image data.
In step S2, the object photographing unit 212 transmits the generated image data of the inspection object to the server device 3a.

In step S3, the identification unit 312 identifies which type of inspection object 4 the inspection object 4 as the subject is by analyzing the image data of the inspection object.
In step S4, the identification unit 312 transmits the identification result (that is, which type of inspection object 4 is) to the wearable terminal 2.

In step S5, the notification unit 214 notifies the user U of the object information corresponding to the identification result. For example, the product name and the image data of the appearance (that is, the packaging container) included in the object information are displayed for notification.

In step S6, the notification unit 214 determines whether or not the identification result by the server device 3a is correct. If the operation to the effect that the identification result is correct is accepted, it is determined as Yes in step S6, and the process proceeds to step S7. On the other hand, if the operation to the effect that the identification result is correct is not accepted, it is determined as No in step S6, the process returns to step S1, and the shooting of step S1 is repeated again.

In step S81, the object information management unit 213 transmits the object information corresponding to the identification result to the server device 3a.
In step S82, the determination unit 315 is set by reading out the machine learning model stored in the learning model storage unit 354. When the machine learning model is used properly according to the type of the inspection object 4, the type of the object information (that is, the determination target this time) is based on the object information received from the wearable terminal 2. The machine learning model corresponding to the type of inspection object 4) is read from the learning model storage unit 354.

In step S83, the determination unit 315 transmits to the wearable terminal 2 that the machine learning model has been set.
In step S10, the notification unit 214 notifies the user U of the status indicating the state of the state determination device 1 as "ready".

In step S11, the notification unit 214 notifies the user U of the image data of the position to be gripped by the gripping mechanism 65 at the time of inspection included in the object information corresponding to the identification result. With the notification in step S11, the user U starts an operation on the operation mechanism 55 in order to grip the inspection object 4.

After that, as described above, the gripping start processing similar to the gripping start processing of the first embodiment described with reference to FIG. 12 is performed. Then, when the gripping start process is completed, the construction process is started. Regarding the step in which the determination unit 114 is the main processing agent in the gripping start processing of the first embodiment, the processing unit 117 is the main processing agent in the gripping start processing of the present embodiment.

Next, referring to FIG. 22, in step S21, the parameter acquisition unit 112 acquires the control parameters related to the current force-tactile sensation calculated in real time.

In step S22, the processing unit 117 determines whether or not the value indicating the force in the control parameter related to the force-tactile sensation acquired in step S21 maintains the first threshold value or more. This determination is a process performed with the intention of confirming that the state in which the inspection object 4 is gripped by the gripping mechanism 65 is properly continued.

When the value indicating the force maintains the first threshold value or more, it is determined as Yes in step S22, and the process proceeds to step S25. On the other hand, when the value indicating the force does not maintain the first threshold value or more (that is, when the value indicating the force is less than the first threshold value), it is determined as No in step S22, and the process proceeds to step S24.

In step S23, the processing unit 117 transmits to the wearable terminal 2 that the failure to grip the inspection object 4 has been detected.

In step S24, the notification unit 214 notifies the user U of the start that indicates the state of the state determination device 1 as an "error and retry". Then, the process returns to step S10 in FIG. 21 and is repeated.

In step S26, the processing unit 117 counts up the value of the counter used in steps S21 to S27 by one. In the construction process, it is not necessary to perform the determination by the state determination device 1, so the step S25 in the first embodiment is not performed.
In step S27, the processing unit 117 determines whether or not the value of the counter has reached the specified count number. When it is reached, it is determined as Yes in step S27, and the process proceeds to step S28. On the other hand, if it has not been reached, it is determined as No in step S27, the process returns to step S21, and the process is repeated. This is a process performed with the intention of acquiring as many control parameters as input data regarding force and tactile sensation.

In step S28, the processing unit 117 stores the control parameters related to force and tactile sensation acquired by repeating step S21 in the parameter storage unit 151. This stored force-tactile control parameter is then utilized as part of the input data for making a damage determination using a machine learning model.

In step S91, the processing unit 117 provides environmental information on the ambient temperature, humidity, and atmospheric pressure of the state determination device 1 measured by the sensor 19 when gripping (for example, when the acquisition of control parameters related to force and tactile sensation is completed). And store it in the environmental information storage unit 154. This stored environmental information is then used as part of the input data for making a determination about damage using a machine learning model.

In step S92, the processing unit 117 transmits the control parameters related to the force and tactile sensation stored in step S28 and the environmental information stored in step S91 as a set to the server device 3a as input data.

In step S93, the determination unit 315 inputs the input data received from the state determination device 1 to the machine learning model set in step S82, and uses the output as the determination result of the determination regarding the damage of the inspection object 4. In this case, the determination result is information indicating whether or not the inspection object 4 is damaged.

In step S94, the determination unit 315 transmits the determination result of whether or not the inspection object 4 is damaged to the wearable terminal 2. Here, the determination result to be transmitted is either "the inspection object 4 is damaged" or "the inspection object 4 is not damaged".

In step S30, the notification unit 214 notifies the user U of the determination result. For example, a text indicating either "the inspection object 4 is damaged" or "the inspection object 4 is not damaged", which is the determination result, is displayed or a voice is output. Or something.
In step S31, the notification unit 214 notifies the user U of the work information. For example, this work information is based on the determination result, and the content is such that the user U specifically instructs the work to be performed next.

The specific examples of the notifications in steps S30 and S31 are the same as the specific examples of the notifications of the first embodiment described with reference to FIGS. 14 and 15.
However, in the first embodiment, a text showing the relationship between the value indicating the position (“measured value” in the figure), which is a control parameter related to force and tactile sensation, and the second threshold value (“threshold value” in the figure) is displayed. Was there. In the second embodiment, instead of this, the relationship between the value indicating the position (“measured value” in the figure), which is a control parameter related to force and tactile sensation, and the value of the feature amount, which is a judgment standard by the machine learning model, is established. You may want to display the text to indicate.
In addition, for example, in the second embodiment, a text indicating a value such as temperature included in the environmental information may be displayed.

After that, as described above, the gripping end processing similar to the gripping end processing of the first embodiment described with reference to FIG. 16 is performed. Then, when the gripping end process is completed, the present state determination process is completed. Regarding the step in which the determination unit 114 is the main processing agent in the gripping end processing of the first embodiment, the processing unit 117 is the main processing agent in the gripping end processing of the present embodiment.

According to the machine learning process and the state determination process described above, objectively based on the control parameters related to the force and tactile sensation when the inspection object 4 is gripped by the state determination device 1 and the environmental information such as temperature, humidity and atmospheric pressure. It is determined that the inspection object 4 is damaged. Therefore, in the present embodiment, as in the first embodiment, the work can be visualized by converting the sensation (here, force and tactile sensation) in the work into data. As a result, the inspection accuracy does not depend on the personal feeling (that is, the subjectivity of the operator), and the inspection can be performed with stable accuracy.
That is, according to the machine learning process and the state determination process of the present embodiment, as in the state determination process of the first embodiment, stable accuracy based on an objective index is used when inspecting for damage. It is possible to realize the inspection in.

[Modification example]
Although the embodiment of the present invention has been described above, this embodiment is merely an example and does not limit the technical scope of the present invention. The present invention can take various other embodiments without departing from the gist of the present invention, and can be modified in various ways such as omission and substitution. In this case, these embodiments and variations thereof are included in the scope and gist of the invention described in the present specification and the like, and are included in the scope of the invention described in the claims and the equivalent scope thereof.
As an example, the embodiment of the present invention described above may be modified as follows.

In the first embodiment and the second embodiment described above, a single server device 3 (or server device 3a) manages object information, identifies the inspection target 4 by image analysis, manages product information, and the like. I was going. Not limited to this, for example, these plurality of functions may be distributed and realized in a plurality of server devices. Further, for example, the state determination device 1 and the wearable terminal 2 may be realized as a single device.

A part or all of the above-mentioned first embodiment and the second embodiment may be combined. For example, until the teacher data for performing the machine learning process is accumulated and the machine learning model is constructed, the state determination process is performed using the second threshold value as in the first embodiment. Then, after the machine learning model is constructed, the state determination process may be performed using the machine learning model as in the second embodiment. In addition, for example, whether the state determination process is performed using the second threshold value as in the first embodiment according to the user's selection operation, the type of the inspection object 4 to be determined, or the like. 2 It may be possible to select whether to perform the state determination process using the machine learning model as in the embodiment.

In the above-mentioned first embodiment, the determination regarding the damage of the inspection object 4 is made based on whether the value indicating the force maintains the first threshold value and the value indicating the position is not less than the second threshold value. Was going. That is, the determination was made based on the change in position (that is, the amount of movement) while the force was constant. Not limited to this, for example, a determination may be made based on a change in force while the position is constant. Alternatively, the determination may be made based on both the change in force and the change in position. That is, if it is possible to appropriately specify the impedance in gripping based on the control parameters related to force and tactile sensation and make a judgment with stable accuracy, the criteria for making the judgment may be appropriately changed.

In the above-mentioned second embodiment, the value indicating the force is maintained at the first threshold value, the value indicating the position is acquired, and the value indicating this position is used as the control parameter related to the force-tactile sensation. Then, using the control parameter (that is, the value indicating the position) related to the force-tactile sensation as a part of the teacher data and the input data, a machine learning model is constructed and a determination regarding the damage of the inspection object 4 is performed. That is, the determination was made based on the change in position (that is, the amount of movement) while the force was constant. Not limited to this, for example, a determination may be made based on a change in force while the position is constant. Alternatively, the determination may be made based on both the change in force and the change in position. That is, if it is possible to appropriately specify the impedance in gripping based on the control parameters related to force and tactile sensation and make a judgment with stable accuracy, the criteria for making the judgment may be appropriately changed.

In the above-mentioned second embodiment, the learning model storage unit 354 is provided in the server device 3a to store the machine learning model, and the determination unit 315 of the server device 3a uses the machine learning model to determine the damage. .. Not limited to this, for example, the state determination device 1 is provided with a storage unit similar to the learning model storage unit 354 to store the machine learning model, and the state determination device 1 is provided with a functional block similar to the determination unit 315 to be in this state. The determination unit of the determination device 1 may use the machine learning model to determine the damage. Alternatively, for example, the wearable terminal 20 is provided with a storage unit similar to the learning model storage unit 354 to store the machine learning model, and the wearable terminal 2 is provided with a functional block similar to the determination unit 315 to provide the determination unit of the wearable terminal 2. May use a machine learning model to make a determination about damage.
As a result, for example, even in an environment where communication with the server device 3a is difficult during the state determination process, it is possible to determine the damage by using the machine learning model.
Further, when using a machine learning method that can secure sufficient computing power in the state determination device 1 or the wearable terminal 2 or can build a machine learning model with a small amount of computation, the state determination device 1 A machine learning unit similar to the machine learning unit 314 may be provided on the wearable terminal 2 or the wearable terminal 2 so that the machine learning unit can also construct the machine learning model.

In the above-mentioned second embodiment, the environmental information is used as a part of the teacher data in the machine learning process and a part of the input data in the state determination process. Not limited to this, for example, the use of environmental information may be omitted. Alternatively, information other than environmental information that indicates an event that affects control parameters related to force and tactile sensation during grasping may be used.

As described above, the state determination system S according to the embodiment of the present invention includes an operation control unit 111, a parameter acquisition unit 112, a determination unit 114, and a notification unit 214.
The operation control unit 111 controls the operation of the state determination device 1 based on the operation of the user, so that the state determination device 1 grips the inspection object 4.
The parameter acquisition unit 112 acquires control parameters related to force and tactile sensation used in the control of the state determination device 1 by the operation control unit 111.
The determination unit 114 determines the damage of the inspection object 4 based on the control parameters related to the force and tactile sensation acquired by the parameter acquisition unit 112.
The notification unit 214 notifies the user of the determination result of the determination unit 114.
As described above, the state determination system S objectively determines the damage of the inspection object 4 based on the control parameters related to the force and tactile sensation when the inspection object 4 is gripped by the gripping mechanism. Therefore, the state determination system S can visualize the work by converting the sensation in the work (here, the force and tactile sensation) into data. As a result, the inspection accuracy does not depend on the personal feeling (that is, the subjectivity of the operator), and the inspection can be performed with stable accuracy. Further, since it has such a configuration, the state determination system S can be easily realized without requiring large-scale equipment such as various sensors and a conveyor for inspection. Therefore, it is possible to apply various products to inspections performed at distribution centers, retail stores, etc. for finding damage during transportation.
That is, according to the state determination system S, when inspecting for damage, it is possible to realize an inspection with stable accuracy based on an objective index.

The inspection object 4 is a packaging container.
The determination unit 114 determines whether or not the contents packaged by the packaging container, which is the inspection target 4, have leaked due to the damage of the packaging container.
This makes it possible to inspect whether or not air leakage or the like has occurred due to damage to the packaging container.

The state determination device 1 includes a master device that accepts a user's operation and a slave device that executes gripping of the inspection object 4.
The motion control unit 111 controls the motion of the state determination device 1 by transmitting control parameters related to force and tactile sensation between the master device and the slave device.
Thereby, the inspection can be performed by using the control parameters related to the force-tactile sensation when controlling the operation of the master device and the operation of the slave device.

The state determination system S further includes an identification unit 312.
The identification unit 312 identifies the type of the inspection object 4 by recognizing the inspection object 4.
The determination unit 114 determines the damage of the inspection object 4 based on the criteria corresponding to the type of the inspection object 4 identified by the identification unit 312.
As a result, the inspection can be performed according to an appropriate standard corresponding to each of the various types of inspection objects 4.

The identification unit 312 identifies the type of the inspection object 4 based on the image of the inspection object 4 taken by the imaging device worn by the user.
Thereby, the type of the inspection object 4 can be more easily identified by the image identification.

The notification unit 214 notifies the user of work instructions based on the determination result together with the determination result of the determination unit 114 or in place of the determination result from the notification device worn by the user.
As a result, it is possible to notify the user of an appropriate work instruction according to the presence or absence of damage or the like.

The determination unit 114 determines the damage of the inspection object 4 based on the position of the predetermined portion included in the state determination device 1 detected based on the control parameters related to the force and tactile sensation.
This allows the inspection to be performed based on a clear criterion of position.

The state determination system S further includes a work information management unit 215 and a work information management unit 313.
The work information management unit 215 and the work information management unit 313 manage the results of work by the user using the state determination device 1.
As a result, not only the inspection but also the management of the work history can be performed.

The motion control unit 111 adjusts the control parameters related to the force and tactile sensation to suppress the gripping of the state determination device 1 to prevent the inspection object 4 from being damaged by the gripping.
This makes it possible to prevent a situation in which the inspection object 4 is accidentally damaged during the inspection.

The operation control unit 111 includes a position sensor, a force / speed allocation conversion block FT ideal force source block FC, and an ideal speed (position) source block PC in order to control the state determination device 1 based on the operation by the user. It includes an inverse conversion block IFT and.
The position sensor detects information about the position accompanying the operation of the state determination device 1.
The force / velocity allocation conversion block FT performs conversion to allocate control energy to energy of a predetermined physical quantity based on information of a predetermined physical quantity corresponding to information regarding a position and information as a reference for control.
The ideal force source block FC and the ideal velocity (position) source block PC calculate a control amount of a predetermined physical quantity based on the energy of a predetermined physical quantity allocated by the force / velocity allocation conversion conversion block FT.
The inverse conversion block IFT reversely converts the control amount to return the output based on the control amount calculated by the ideal force source block FC and the ideal speed (position) source block PC to the state determination device 1, and the state determination device 1 Determine the input to.
As a result, when each user performs an operation accompanied by a force-tactile sensation, the operation of the state determination device 1 can be controlled based on the control parameters related to the force-tactile sensation.

As described above, the state determination system S according to the embodiment of the present invention includes the master side actuator 52, the slave side actuator 62, the operation control unit 111, the determination unit 315 or the determination unit 114, and the notification unit 214. To prepare for.
The master side actuator 52 operates the operation mechanism 55.
The slave side actuator 62 operates the gripping mechanism 65.
The motion control unit 111 controls the operation of the gripping mechanism 65 to grip the inspection object 4 by driving the slave side actuator 62 in response to the user's operation input to the operating mechanism 55, and the gripping mechanism 65 controls the motion of gripping the inspection object 4. By driving the actuator 52 on the master side in response to the input reaction force from the inspection object 4, the operation mechanism 55 controls the operation of transmitting the reaction force to the user, that is, the operation control is performed.
The determination unit 315 or the determination unit 114 determines whether the inspection object 4 is damaged based on the control parameters used by the operation control unit 111 in the operation control.
The notification unit 214 notifies the user of the determination result of the determination unit 315.
As described above, the state determination system S is objectively based on the control parameters corresponding to the user's operation and the reaction force (that is, force tactile sensation) from the inspection object 4 when the inspection object 4 is gripped by the gripping mechanism. It is determined that the inspection object 4 is damaged. Therefore, the state determination system S can visualize the work by converting the sensation in the work (here, the force and tactile sensation) into data. As a result, the inspection accuracy does not depend on the personal feeling (that is, the subjectivity of the operator), and the inspection can be performed with stable accuracy. Further, since it has such a configuration, the state determination system S can be easily realized without requiring large-scale equipment such as various sensors and a conveyor for inspection. Therefore, it is possible to apply various products to inspections performed at distribution centers, retail stores, etc. for finding damage during transportation.
That is, according to the state determination system S, when inspecting for damage, it is possible to realize an inspection with stable accuracy based on an objective index.

The determination unit 315 operates the operation control unit 111 when gripping the inspection object 4 to be determined with respect to the machine learning model in which the relationship between the control parameter and the determination result regarding the damage of the inspection object 4 is machine-learned. Judgment is made by inputting the control parameters used in the control.
As a result, the judgment is made based on an objective index called a machine learning model in which the relationship between the control parameter and the judgment result regarding the damage of the inspection object 4 is machine-learned, and the inspection with more stable accuracy is realized. can do.

The state determination system S further includes a machine learning unit 314.
The machine learning unit 314 performs machine learning using a set of the control parameter and the determination result regarding the damage of the inspection object 4 as teacher data, and thereby performs the relationship between the control parameter and the determination result regarding the damage of the inspection object 4. Build a machine learning model that is machine-learned.
As a result, it is possible to construct an objective index called a machine learning model in which the relationship between the control parameter and the determination result regarding the damage of the inspection object 4 is machine-learned.

The machine learning model is machine-learned not only about the relationship between the control parameters and the determination result regarding the damage of the inspection object 4, but also about the relationship with the surrounding environment when the master side actuator 52 and the slave side actuator 62 are driven. It is a machine learning model.
This makes it possible to use a machine learning model that takes into account more detailed information such as the surrounding environment when the master side actuator 52 and the slave side actuator 62 are driven.

The determination result regarding the damage of the inspection object 4 used in machine learning is the determination result made by the user who input the operation to the operation mechanism 55 when grasping the inspection object 4.
Thereby, for example, a machine learning model based on an accurate determination result by a skilled user or a user who knows the correct answer in advance can be used.

The state determination system S further includes an identification unit 312.
By recognizing the inspection object 4 based on the image of the inspection object 4 taken by the photographing device worn by the user, the type of the inspection object 4 is identified.
The determination unit 315 or the determination unit 114 determines the damage of the inspection object 4 based on the criteria corresponding to the type of the inspection object 4 identified by the identification unit 312.
Thereby, not only the type of the inspection object 4 can be more easily identified by the image identification, but also the inspection can be performed according to an appropriate standard corresponding to each of various types of inspection objects 4.

The notification unit 214 notifies the user of work instructions based on the determination result together with the determination result of the determination unit 315 or the determination unit 114, or in place of the determination result, from the notification device worn by the user.
As a result, it is possible to notify the user of an appropriate work instruction according to the presence or absence of damage or the like.

[Realization of functions by hardware and software]
The function of executing a series of processes according to the above-described embodiment can be realized by hardware, software, or a combination thereof. In other words, it suffices if the function of executing the above-mentioned series of processes is realized in any one of the state determination systems S, and the mode in which this function is realized is not particularly limited.

For example, when the function of executing the above-mentioned series of processes is realized by a processor that executes arithmetic processing, the processor that executes this arithmetic processing is composed of various processing units such as a single processor, a multiprocessor, and a multicore processor. In addition to the above, the present invention includes a combination of these various processing units and a processing circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

Further, for example, when the function of executing the above-mentioned series of processes is realized by software, the programs constituting the software are installed in the computer via a network or a recording medium. In this case, the computer may be a computer having dedicated hardware built-in, or a general-purpose computer capable of performing a predetermined function by installing a program (for example, a general-purpose personal computer or the like). It may be an electronic device in general). Further, the step for describing the program may include only the processes performed in time series in the order thereof, but may include the processes executed in parallel or individually. Further, the steps for describing the program may be executed in any order within a range that does not deviate from the gist of the present invention.

The recording medium on which such a program is recorded may be provided to the user by being distributed separately from the computer main body, or may be provided to the user in a state of being incorporated in the computer main body in advance. In this case, the storage medium distributed separately from the computer itself is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disc is composed of, for example, a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versaille Disc), a Blu-ray (registered trademark) Disc (Blu-ray disc), or the like. The magneto-optical disk is composed of, for example, MD (MiniDisc) or the like. These storage media are attached to, for example, the drive 18 of FIG. 8 and the drive 38 of FIG. 10, and are incorporated in the computer body. The recording media provided to the user in a state of being incorporated in the computer body in advance include, for example, the ROM 12 of FIG. 8 in which the program is recorded, the ROM 22 of FIG. 9, the ROM 32 of FIG. 10, and the storage unit 15 of FIG. It is composed of a hard disk or the like included in the storage unit 25 of FIG. 9 or the storage unit 35 of FIG.

1 Status determination device, 2 Wearable terminal, 3,3a server device, 4 Inspection target, 10,10a control unit, 11,21,31 processor, 12,22,32 ROM, 13,23,33 RAM, 14,24 , 34 communication unit, 15, 25, 35 storage unit, 16, 26, 36 input unit, 17, 27, 37 output unit, 18, 38 drive, 28 camera, 29, 19 sensor, 50 master side unit, 51 master side Driver, 52 Master side actuator, 53 Master side position sensor, 55 Operation mechanism, 60 Slave side unit, 61 Slave side driver, 62 Slave side actuator, 63 Slave side position sensor, 65 Grip mechanism, 111 Operation control unit, 112 Parameter acquisition Unit, 113 Judgment standard setting unit, 114 Judgment unit, 115 Correct answer information acquisition unit, 116 Environmental information acquisition unit, 117 Processing unit, 151 Parameter storage unit, 152 Judgment standard storage unit, 153 Correct answer information storage unit, 154 Environmental information storage unit , 211 Instruction interpretation unit, 212 Object photography unit, 213,311 Object information management unit, 214 Notification unit, 215, 313 Work information management unit, 251, 351 Object information storage unit, 252, 352 Work information storage unit, 312 Identification unit, 314 Machine learning unit, 315 Judgment unit, 353 Teacher data storage unit, 354 Learning model storage unit, CS control target system, FT force / speed allocation conversion block, FC ideal power source block, PC ideal speed (position) Source block, IFT inverse conversion block, N network, S status determination system, U user

Claims (9)

The actuator on the operation mechanism side that operates the operation mechanism, and
The actuator on the gripping mechanism side that operates the gripping mechanism,
By driving the gripping mechanism side actuator in response to the user's operation input to the operating mechanism, the gripping mechanism controls the operation of gripping the inspection target, and the inspection target input to the gripping mechanism. An operation control means for performing an operation control in which the operation mechanism controls an operation of transmitting the reaction force to the user by driving the actuator on the operation mechanism side in response to a reaction force from an object.
A determination means for determining damage to the inspection object based on the control parameters used by the operation control means in the operation control, and a determination means.
A notification means for notifying the user of the determination result of the determination means, and
A state determination system characterized by being equipped with. The determination means is a machine learning model in which the relationship between the control parameter and the determination result regarding the damage of the inspection object is machine-learned. The determination is made by inputting the control parameters used in the operation control.
The state determination system according to claim 1. By performing machine learning using the set of the control parameter and the determination result regarding the damage of the inspection object as teacher data, the relationship between the control parameter and the determination result regarding the damage of the inspection object is machine-learned. Machine learning means to build a machine learning model,
The state determination system according to claim 1 or 2, further comprising. In the machine learning model, in addition to the relationship between the control parameter and the determination result regarding the damage of the inspection object, the relationship with the surrounding environment when the actuator on the operation mechanism side and the actuator on the gripping mechanism side are driven is also obtained. It is a machine learning model that has been machine-learned.
The state determination system according to claim 2 or 3, wherein the state determination system is characterized in that. The determination result regarding the damage of the inspection object used in the machine learning is the determination result made by the user who input the operation to the operation mechanism at the time of grasping the inspection object.
The state determination system according to any one of claims 2 to 4, wherein the state determination system is characterized by the above. Further provided with an identification means for identifying the type of the inspection object by recognizing the inspection object based on the image of the inspection object taken by the imaging device worn by the user.
The determination means determines the damage of the inspection object based on the criteria corresponding to the type of the inspection object identified by the identification means.
The state determination system according to any one of claims 1 to 5, wherein the state determination system is characterized in that. The notification means notifies the user of a work instruction based on the determination result together with the determination result of the determination means or instead of the determination result from the notification device worn by the user.
The state determination system according to any one of claims 1 to 5, wherein the state determination system is characterized in that. The actuator on the operation mechanism side that operates the operation mechanism, and
The actuator on the gripping mechanism side that operates the gripping mechanism,
It is a state judgment method performed by a system equipped with
By driving the gripping mechanism side actuator in response to the user's operation input to the operating mechanism, the gripping mechanism controls the operation of gripping the inspection target, and the inspection target input to the gripping mechanism. An operation control step that controls an operation in which the operation mechanism controls an operation of transmitting the reaction force to the user by driving the actuator on the operation mechanism side in response to a reaction force from an object.
A determination step for determining damage to the inspection object based on the control parameters used in the operation control of the operation control step, and a determination step.
A notification step for notifying the user of the determination result of the determination step,
A state determination method comprising. By driving the actuator on the gripping mechanism side that operates the gripping mechanism in response to the user's operation input to the operating mechanism, the gripping mechanism controls the operation of gripping the object to be inspected and is input to the gripping mechanism. By driving the actuator on the operation mechanism side that operates the operation mechanism in response to the reaction force from the inspection object, the operation control that the operation mechanism controls the operation of transmitting the reaction force to the user is performed. Motion control function and
A determination function for determining damage to the inspection object based on the control parameters used by the operation control function in the operation control, and a determination function.
A notification function for notifying the user of the determination result of the determination function, and
A state judgment program characterized by realizing a computer.

Images