JP6736298B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

The present invention relates to a technique for assisting an operator in learning an appropriate work by training an operation using a simulated tool imitating a tool.

Conventionally, in the field of manufacturing industry, a training tool using a mixed reality (MR) system in which a real space and a virtual space are seamlessly combined has been introduced. This is for a worker to learn an appropriate work through training for operating a virtual object by using a simulated tool that imitates a tool. As one of the image display means, a head mounted display (hereinafter, referred to as HMD: head mounted display) in which an image pickup device such as a video camera and a display are integrated is used. According to the mixed reality system, the target object can be represented by CG (computer graphics), and can be displayed on the HMD by superimposing the image on the real world. As a result, the worker can experience the work while checking as if the operation area exists there. In this experience, it is possible to repeat many times without having to make and repair the actual parts.

A worker in a manufacturing site can use a training tool of a mixed reality system using a simulated tool to learn the work. By doing so, you can learn the work efficiently. For example, teaching by visual information presentation can be provided by displaying the contents to be carried out about the work procedure and the attention information that calls attention on the display screen of the HMD worn by the worker. Further, by holding a simulated tool equipped with a vibrating device and applying a vibration stimulus according to the position and orientation of the simulated tool to the worker, it is possible to present information accompanied by a sensation.

Patent Document 1 discloses a technique of feeding back tactile information to an operator according to the position and orientation of a body part. Patent Document 2 discloses a technique for controlling the degree of force sense to be presented according to the degree of fastening in the work of fastening a virtual screw object to a virtual screw hole object using a simulated tool attached with a device that presents force sense. There is.

JP2012-74076A JP, 2008-55527, A

However, in training in which a worker operates a virtual object using a simulated tool that imitates a tool, there is a problem that it is difficult to learn an accurate work like a skilled worker.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for assisting an operator in learning an appropriate work.

Therefore, the present invention is an information processing apparatus, which includes a status acquisition unit that acquires status information indicating at least one of the position and the attitude of the simulated tool at the time of operation by a user , and time-series information of the status information. based on the trajectory generating means for generating a tool path information indicating the trajectory of the state of the simulated tool, and the tool path information was stored in the memory means, shows the standard locus of the state of the simulated tool, the reference It is characterized by further comprising a stimulus control means for controlling a stimulus generation device for generating a stimulus to the user based on the reference trajectory information whose range varies according to time .

According to the present invention, it is possible to provide a technique that assists an operator in learning an appropriate work.

It is a general view of a support system. It is a figure which shows the hardware constitutions of an information processing apparatus, a simulation tool, and HMD. It is a figure which shows the software structure of an information processing apparatus. It is a figure which shows an example of tool locus information. It is a flow chart which shows information processing. It is explanatory drawing of information processing.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is an overall view of a support system as an example of an information processing system according to this embodiment. The support system includes an information processing device 100, a simulation tool 110, and an HMD 120. The worker A wears the HMD 120. A virtual object 130 is displayed by the HMD 120. The worker A experiences the work in the mixed reality space in which the virtual object 130 is projected by the HMD 120. In the present embodiment, an operation of applying a constant coating film to the virtual object 130 by the simulation tool 110 simulating an electric spray will be described as an example. The simulation tool 110 can generate a stimulus for teaching when the work of the worker is not appropriate. The information processing apparatus 100 controls the HMD 120 such as generating a video of a virtual object based on the information received from the simulation tool 110 and the HMD 120. The information processing apparatus 100 also controls the generation of stimulation by the simulation tool 110.

FIG. 2A is a diagram showing a hardware configuration of the information processing device 100. The information processing device 100 includes a CPU 201, a ROM 202, a RAM 203, an HDD 204, a display unit 205, an input unit 206, and a communication unit 207. The CPU 201 reads out the control program stored in the ROM 202 and executes various processes. The RAM 203 is used as a main memory of the CPU 101 and a temporary storage area such as a work area. The HDD 204 stores various information such as image data and various programs. The display unit 205 displays various information. The input unit 206 has a keyboard and a mouse and receives various operations by the user. The communication unit 207 connects to the simulation tool 110 and the HMD 120 by short-range wireless communication such as Bluetooth. The functions and processing of the information processing apparatus 100, which will be described later, are realized by the CPU 201 reading a program stored in the ROM 202 or the HDD 204 and executing the program.

FIG. 2B is a diagram showing a hardware configuration of the simulation tool 110. The simulation tool 110 includes a detection unit 211, a stimulus generation unit 212, and a communication unit 213. The detection unit 211 detects the state of the simulated tool 110. Here, the state is the position and orientation of the simulated tool 110. Note that, as another example, the state may be at least one of the position and the posture of the simulated tool 110. The detection unit 211 is, for example, a 6-DOF position/orientation sensor. The stimulus generator 212 generates a stimulus to be given to the worker who is the user of the simulation tool 110. The stimulus generation unit 212 specifically generates sensory stimuli such as hearing, touch, and force. For the stimulus generator 212, for example, an eccentric motor (ERM), a linear vibrator (LRA), a piezo-piezoelectric element or the like may be used as a device for generating a vibration stimulus. Further, in the stimulus generator 212, an electrodynamic speaker, an electrostatic speaker, a piezoelectric speaker, or the like may be used as a device for generating an auditory stimulus. The stimulus generator 212 is an example of a stimulus generator. The communication unit 213 connects to the information processing device 100 by short-range wireless communication.

FIG. 2C is a diagram showing the hardware configuration of the HMD 120. The HMD 120 has an imaging unit 221, a display unit 222, and a communication unit 223. The imaging unit 221 captures an image in the line-of-sight direction of the wearer. The display unit 222 displays a virtual object. The display unit 222 displays the composite image in which the virtual object is composited on the captured image obtained by the stimulus generation unit 212, so that the wearer can experience the mixed reality space. The communication unit 223 connects to the information processing device 100 by short-range wireless communication.

FIG. 3 is a diagram showing a software configuration of the information processing apparatus 100. The information processing device 100 includes an information acquisition unit 301, a trajectory generation unit 302, a tool trajectory storage unit 303, a reference trajectory storage unit 304, a deviation determination unit 305, and a stimulation control unit 306. The information processing device 100 further includes an image acquisition unit 307, a model storage unit 308, an image generation unit 309, and a display control unit 310.

The information acquisition unit 301 acquires, from the simulated tool 110 via the communication unit 207, state information indicating the state of the simulated tool 110 when the operator operates the simulated tool 110, which is detected by the detection unit 211 of the simulated tool 110. To do. The trajectory generation unit 302 generates tool trajectory information indicating a tool trajectory that is a trajectory of the state of the simulated tool 110 based on the state information, and stores the tool trajectory information in the tool trajectory storage unit 303. FIG. 4 is a diagram showing an example of the tool trajectory information 400. The locus generation unit 302 associates the state information acquired by the information acquisition unit 301 with the time when the state information was input. In the example of FIG. 4, the time is based on the start of work. The trajectory generation unit 302 generates a set of state information and time as one record, and arranges the records in time series to generate state information along the time series, that is, time series information of the state information as the tool trajectory information 400. To do. The state information indicates position coordinates X, Y, Z and rotation angle coordinates Qx, Qy, Qz of the simulated tool 110 in the mixed reality space.

Returning to FIG. 3, the reference trajectory storage unit 304 stores the reference trajectory information in advance. Here, the reference trajectory information is information indicating an appropriate trajectory obtained in the work of the simulated tool 110. It is assumed that the reference trajectory information is generated in advance based on state information when a skilled person has previously performed a work experience on a virtual object using the simulated tool 110, and is stored in the tool trajectory storage unit 303. .. The data configuration of the reference trajectory information stored in the reference trajectory storage unit 304 is the same as the data configuration of the tool trajectory information 400.

The reference trajectory storage unit 304 further stores a reference range in advance. Here, the reference range is information indicating an allowable range (numerical value range) at each time, based on the three-dimensional coordinates and the three-dimensional rotation coordinates of the reference trajectory. If the tool path is within this reference range, it is determined that the work is being performed properly, and if the tool path exceeds the reference range, the work needs to be corrected and the operator is notified. It The width of the reference range is assumed to be different depending on the time of day and the work content. This is because the allowable threshold for coordinate deviation may differ depending on the difference in the density of work areas, the level of risk, and the like.

The deviation determining unit 305 sequentially compares the state information of the tool trajectory information and the reference trajectory information at the same time, and determines whether the tool trajectory indicated by the tool trajectory information deviates from the reference trajectory. Specifically, the deviation determining unit 305 confirms whether or not the tool locus is included in a reference range based on the reference locus, and if the tool locus deviates from the reference range, the tool locus deviates from the reference locus. judge. When the deviation determining unit 305 further determines that the tool path deviates from the reference path, the deviation determining unit 305 calculates the deviation degree of the tool path. Here, the deviation degree is an index value indicating the degree to which the tool path deviates from the reference. The deviation determination unit 305 calculates the difference between the tool trajectory and the boundary position of the reference range as the deviation degree. The stimulation control unit 306 controls the stimulation generation unit 212 of the simulated tool 110 according to the determination result of the deviation determination unit 305.

On the other hand, the image acquisition unit 307 acquires, from the HMD 120 via the communication unit 207, a captured image of the physical space in which the simulated tool 110 is present, obtained by the imaging unit 221 of the HMD 120 (image acquisition process). The model storage unit 308 stores a virtual model for displaying the virtual object 130. The image generation unit 309 generates a composite image in which a virtual model or the like is combined with the captured image acquired by the image acquisition unit 307. The display control unit 310 controls the display by the HMD 120. The display control unit 310 controls the HMD 120 such that the composite image generated by the image generation unit 309 is displayed on the display unit 222 of the HMD 120, for example.

FIG. 5 is a flowchart showing information processing by the information processing device 100. In S500, the information acquisition unit 301 acquires state information from the simulated tool 110 (state acquisition process). Next, in S501, the trajectory generation unit 302 generates tool trajectory information by associating the state information acquired in S500 with the time when the state information was input (trajectory information generation processing). Then, the trajectory generation unit 302 records the tool trajectory information in the tool trajectory storage unit 303. Next, in S502, the deviation determination unit 305 compares the tool trajectory information at the corresponding time with the reference trajectory information stored in the reference trajectory storage unit 304, so that the tool trajectory indicated by the tool trajectory information is in the reference range. Is included in. If the tool locus is included in the reference range (Yes in S502), the deviation determining unit 305 advances the process to S508. If the tool path exceeds the reference range (No in S502), the deviation determination unit 305 advances the process to S503.

In S503, the deviation determination unit 305 calculates the difference between the tool trajectory and the boundary position of the reference range as the deviation degree. Next, in S504, the deviation determination unit 305 determines the strength of the stimulus according to the deviation degree. In this embodiment, it is assumed that two range levels are defined for the degree of deviation, and the stimulation intensity is associated with each range level and stored in the HDD 204 or the like, for example. Then, the departure determining unit 305 identifies the range level to which the departure degree calculated in S503 belongs, and identifies the strength of the stimulus associated with the identified range level. The degree of deviation and the number of range levels that are the boundaries of each range level are not particularly limited, and must be a sufficient number range that allows the operator to identify the difference in stimulation intensity and stimulation pattern. Is preferred.

Next, in S505, the stimulation control unit 306 controls the stimulation generation unit 212 of the simulated tool 110 so as to generate the stimulation having the intensity determined in S504 (stimulation control process). Specifically, the deviation determination unit 305 transmits an instruction to the simulated tool 110 via the communication unit 207 so as to generate the stimulus having the intensity determined in S504. In the simulated tool 110, when the instruction is received, the stimulus generation unit 212 generates a stimulus according to the instruction. For example, when the departure degree belongs to the lowest range level, the departure determination unit 305 determines the output having the lowest voltage pulse width as the stimulation intensity. Then, in response to this, the stimulus generator 212 of the simulation tool 110 generates the weakest stimulus. Further, for example, when the departure degree belongs to the highest range level, the departure determination unit 305 determines the output having the strongest voltage pulse width as the stimulation intensity. Then, in response to this, the stimulus generator 212 generates the strongest stimulus.

Next, in step S506, the image generation unit 309 generates a composite image in which the warning message and the guidance message are combined with the captured image according to the intensity range level determined in step S504. Here, the warning message and the guidance message are examples of presentation information. The processing of S506 is an example of image generation processing for generating a composite image. In the present embodiment, it is assumed that a message image to be combined is stored in the HDD 204 or the like in association with each range level. Then, the image generation unit 309 combines the message image associated with the range level with the captured image. Next, in S507, the display control unit 310 controls the display unit 222 of the HMD 120 to display the composite image generated in S506. Specifically, the display control unit 310 transmits an instruction to the HMD 120 via the communication unit 207 to display the composite image generated in S506. When the HMD 120 receives the instruction, the display unit 222 displays the combined image according to the instruction. Thereby, a visual stimulus can be given to the worker.

For example, when the deviation degree belongs to the lowest range level, the image generation unit 309 synthesizes a warning message with the captured image. Then, in response to this, the display unit 222 of the HMD 120 displays the composite image of the warning message. Further, for example, when the deviation degree belongs to the highest range level, in addition to the warning message, the arrow CG is combined in the position/orientation direction to be corrected. As a result, more specific teaching can be given to the operator and the proper use of the simulated tool 110 can be prompted.

Next, in S508, the CPU 201 confirms whether or not the work by the worker using the simulated tool 110 is completed. It is assumed that the CPU 201 determines that the work is completed when the notification of the work completion is received from the simulation tool 110. When the work is completed (Yes in S508), the CPU 201 ends the process. When the work is not completed (No in S508), the CPU 201 advances the process to S500 and continues the process.

Next, the work training will be specifically described with reference to FIG. As shown in FIG. 6A, the worker A wears the HMD 120 and moves the simulated tool 110 while spraying the virtual paint using the simulated tool 110 simulating an electric spray. Here, the work of applying a coating film having a constant thickness to the virtual object 130 is performed. It is assumed that the work starts spraying from the start point P0, sprays from the upper part to the lower part of the virtual object 130 while moving the simulated tool 110 to the left and right, and ends the work at the end point P20. In the present embodiment, the image generation unit 309 of the information processing device 100 generates a composite image in which the reference trajectory from the start point P0 to the end point P20 is further superimposed on the virtual object 130, and the display unit 222 of the HMD 120 displays It is assumed that the composite image on which the reference trajectory is superimposed is displayed. The virtual object 130, the simulated tool 110, and the HMD 120 are treated with the common virtual space coordinates 600 in the mixed reality system as a reference.

FIG. 6B is a diagram in which the Y coordinate (thick broken line) of the simulation tool 110 in the virtual space coordinate 600 and the voltage waveform (thick solid line) output by the stimulation generation unit 212 are plotted with respect to the time axis. .. P0, P1,..., P7 in FIG. 6B correspond to P0, P1,..., P7 in FIG. 6A. The reference range is created based on the trajectory of the simulated tool 110 by a skilled worker. The reference range may change depending on the time, and in the example of FIG. 6A, the reference range in the work range of P4 to P6 is narrower than the reference range of P0 to P3.

In the example of FIG. 4B, the worker A can start the spraying from the start point P0 and can operate the simulated tool 110 within the Y reference range from P2 to P2. During this time, the pulse voltage is applied to the stimulus generator 212 with the minimum pulse width. In the working range of P2 to P3, the tool locus deviates from the Y reference locus. Here, the deviation degree corresponds to the distance in the vertical direction from the region boundary of the Y reference range in FIG. 6B to the thick broken line indicating the trajectory of the simulated tool 110. The pulse width of the applied voltage is controlled according to the degree of deviation. According to the magnitude of the pulse width, the vibration stimulus that the worker A receives through the simulated tool 110 becomes large or small. With this mechanism, the worker A can experience in real time how far he/she is operating the simulated tool 110 from the deviation allowance while performing a series of operations.

In the work range from P4 to P6, which is the turning point of the work, the tool locus is outside the Y reference range. Therefore, even in this range, the vibration stimulus corresponding to the deviation degree is presented. After P6, since the tool locus is within the Y reference range, the pulse voltage is applied with the minimum pulse width, and the vibration stimulus is presented accordingly.

FIG. 6C is a diagram in which the vertical rotation coordinates Qz (thick broken line) of the simulated tool 110 in the virtual space coordinates 600 are plotted with respect to the time axis. Further, FIG. 6D is a diagram showing a display example of the display unit 222 of the HMD 120. At the starting points P0 to P3, the worker A operates the simulated tool 110 within the Qz reference range, but the tool locus deviates from the Y reference range in the work range of P4 to P6, which is the turning point of the work. ing. In this way, when the tool path deviates from the Y reference range, as shown in FIG. 6D, a captured image in which a warning message 611 such as “Let's tilt in the direction of the arrow” and the arrow image 612 are combined with the virtual object. Is displayed on the display unit 222. The display control unit 310 may further control to display a message prompting the worker A to return to the correct trajectory depending on the degree of deviation. If the tool path is within the Y reference range, the warning message 611 and the arrow image 612 are not displayed.

As described above, according to the present embodiment, the worker increases the strength of the sensory stimulus such as the tactile stimulus by the vibration or the auditory stimulus by the sound through the training for operating the virtual object using the simulated tool 110 and the HMD 120. It can be perceived including. By presenting such sensory stimuli by the training tool, the worker can grasp the work place to be corrected by him/herself in real time and learn an appropriate work in the same manner as a skilled worker. That is, the support system according to the present embodiment can provide a technique for assisting a worker in learning an appropriate work.

Next, a modified example of the first embodiment will be described. As a first modified example, in the present embodiment, the simulated tool 110 detects the state of the simulated tool 110, but the subject of state detection is not limited to the simulated tool 110. As another example, the HMD 120 may detect the state based on the captured image. Specifically, it is assumed that the simulated tool 110 is attached with an index for state detection. Then, the HMD 120 may capture an image of the simulated tool 110, and specify the state of the simulated tool 110 based on the correspondence relationship between the in-image coordinates of the index in the obtained captured image and the three-dimensional coordinates. Moreover, the information processing apparatus 100 may acquire a captured image from the HMD 120 and specify the state of the simulated tool 110 from the captured image. Further, the information processing apparatus 100 may specify the state from the captured image, acquire the state information from the simulated tool 110, and combine the information to obtain the state information as described in the embodiment. Good.

As a second modification, the tool locus information and the reference locus information may include not only the locus information, but also moving speed information obtained by differentiating each locus information with respect to time. Further, in this case, the deviation determining unit 305 also compares the moving speeds in the comparison between the tool trajectory and the reference trajectory. Then, the deviation determination unit 305 considers deviation of the moving speed (speed and angular speed) of the simulated tool 110 of the worker from the moving speed (speed and angular speed) related to the reference trajectory based on the comparison result, and comprehensively. Degree of deviation should be calculated.

Further, the tool locus information and the reference locus information may include moving acceleration information obtained by subjecting each locus information to second-order differentiation with respect to time, instead of moving speed. Further, in this case, the deviation determination unit 305 also compares the moving accelerations in the comparison between the tool trajectory and the reference trajectory. Then, the deviation determination unit 305 considers the deviation of the moving acceleration (acceleration and angular acceleration) of the simulated tool 110 of the worker from the moving acceleration (acceleration and angular acceleration) related to the reference trajectory based on the comparison result. The total deviance shall be calculated.

As a third modification, the stimulus control unit 306 may control the stimulus generation unit 212 so that different stimuli are generated according to the deviation degree, and specific control relating to stimulus generation is performed in the embodiment. It is not limited. For example, the stimulation control unit 306 may control the stimulation generation unit 212 to increase the stimulation intensity in proportion to the departure degree instead of dividing the departure degree into a plurality of stages. Further, as another example, the stimulus control unit 306 may also control to generate a stimulus according to the degree of deviation even within the reference range. Further, the stimulation control unit 306 may change the type of stimulation in place of the intensity of stimulation depending on the degree of deviation. For example, the stimulation control unit 306 may control the length of the vibration time of the intermittent vibration of the stimulation. Further, the stimulation control unit 306 may control the level of the vibration frequency.

As a fourth modification, the simulation tool 110 may include a speaker as a stimulus generator. Then, in this case, the stimulus control unit 306 may control the size of the buzzer sound from the speaker, or may control the high and low of the frequency of the buzzer sound according to the departure degree, and the buzzer sound pattern. May be controlled to be changed.

As a fifth modified example, the process performed by each device of the support system, particularly the process performed by the information processing device 100 may be performed by any device in the support system, and the main device thereof is limited to the embodiment. It is not something that will be done. For example, the simulated tool 110 may generate the tool trajectory information.

As a sixth modification, the support system only needs to support the training using the simulated tool 110, and the support target is not limited to the training in the mixed reality space.

Although the preferred embodiments of the present invention have been described above in detail, the present invention is not limited to the specific embodiments, and various modifications are possible within the scope of the gist of the present invention described in the claims. -Can be changed.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Information Processing Device 110 Simulated Tool 120 HMD

Claims (13)

A state acquisition unit that acquires state information indicating the state of at least one of the position and the posture of the simulated tool at the time of operation by the user,
Trajectory generating means for generating tool trajectory information indicating a trajectory of the state of the simulated tool, based on time-series information of the state information,
Said tool path information was stored in the memory means, wherein indicates the reference trajectory of the state of the simulated tool, the reference locus information reference range varies according to the time, based on the occurrence of irritation to the user And a stimulus control means for controlling the stimulus generator. The information processing apparatus according to claim 1, wherein the trajectory generation means generates movement speed information in the time-series information of the state information as the tool trajectory information. The information processing apparatus according to claim 1, wherein the trajectory generation means generates movement acceleration information in the time-series information of the state information as the tool trajectory information. The stimulation control unit, an information processing apparatus according to any one of claims 1 to 3, wherein the controller controls so as to generate a stimulation based on a difference between the tool locus information and the reference trajectory information. The stimulation control unit, when the difference between the tool locus information and the reference locus information exceeds the reference range, according to claim 4, wherein the controller controls the stimulus generator to generate said stimulus Information processing device. The information according to claim 5 , wherein the stimulus control means controls to generate a stimulus according to the difference when the difference between the tool trajectory information and the reference trajectory information exceeds a reference range. Processing equipment. Image acquisition means for acquiring a captured image of the physical space in which the simulated tool exists,
Image generating means for generating a combined image in which presentation information to be presented to the user is combined with the captured image,
It said tool path information, and the reference locus information, on the basis of any one of claims 1 to 6, further comprising a display control means for controlling the display on the display means of the composite image The information processing device according to 1. The information processing apparatus according to claim 7 , wherein the image generation unit generates the combined image in which a warning message is combined as the presentation information. The information processing apparatus according to claim 7 , wherein the image generation unit generates the combined image in which the reference trajectory of the simulated tool is combined as the presentation information. A simulated tool operated by the user,
A stimulus generator for generating a stimulus to the user;
A status acquisition unit that acquires status information indicating at least one of the position and the attitude of the simulated tool,
Trajectory generating means for generating tool trajectory information indicating a trajectory of the state of the simulated tool, based on time-series information of the state information,
Said tool path information was stored in the memory means, wherein indicates the reference trajectory of the state of the simulated tool, the reference locus information reference range varies according to the time, based on the control of the stimulus generating device An information processing system, comprising: The information processing system according to claim 10 , wherein the stimulus generator is provided in the simulated tool. An information processing method executed by an information processing device, comprising:
A state acquisition step of acquiring state information indicating at least one of the position and orientation of the simulated tool at the time of operation by the user,
A trajectory generation step of generating tool trajectory information indicating a trajectory of the state of the simulated tool, based on time-series information of the state information,
Said tool path information was stored in the memory means, wherein indicates the reference trajectory of the state of the simulated tool, the reference locus information reference range varies according to the time, based on the occurrence of irritation to the user And a stimulus control step for controlling the stimulus generator. Computer,
A state acquisition unit that acquires state information indicating the state of at least one of the position and the posture of the simulated tool at the time of operation by the user,
Trajectory generating means for generating tool trajectory information indicating a trajectory of the state of the simulated tool, based on time-series information of the state information,
Said tool path information was stored in the memory means, wherein indicates the reference trajectory of the state of the simulated tool, the reference locus information reference range varies according to the time, based on the occurrence of irritation to the user A program for functioning as a stimulus control means for controlling the stimulus generator.

Images