JP2016062177A - Controller, information processing device, and robot system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a controller that can provide notification of the state of an image processing device.SOLUTION: A controller comprises a reception unit that receives, from an image processing device for processing a captured image, state information of the image processing device, a determination unit that determines, on the basis of the state information, a state of the image processing device, and a generation unit that generates a transmission signal to be transmitted to an external device on the basis of a result of determination by the determination unit.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a controller, an information processing apparatus, and a robot system.

  Research and development have been conducted on a method of calculating a position and orientation of a work target based on a captured image captured by an imaging unit, and causing a robot to perform a predetermined operation based on the calculated position and orientation.

  In this regard, an image processing device that is connected to a control device that controls a robot and is separate from a control device that performs image processing such as calculation of the position and orientation of a work target based on a captured image is known. w (see Non-Patent Document 1).

http://robots.epson.com/admin/uploads/product_catalog/files/EPSON_CV1_Vision%20(RevB).pdf `` Vision Guidance for Epson Robots ''

  However, the conventional control device cannot detect the state even when some abnormality occurs in the hardware of the image processing device.

  Accordingly, the present invention has been made in view of the above-described problems of the prior art, and provides a controller, an information processing apparatus, and a robot system that can notify the state of an image processing apparatus.

According to one aspect of the present invention, a receiving unit that receives state information of the image processing device from an image processing device that processes a captured image, a determination unit that determines the state of the image processing device based on the state information, A controller including a generation unit that generates a transmission signal to be transmitted to an external device based on a determination result by the determination unit.
With this configuration, the controller receives the status information of the image processing device from the image processing device that processes the captured image, determines the status of the image processing device based on the status information, and sends to the external device based on the determination result. A transmission signal to be transmitted is generated. Thereby, it is possible to notify the state of the image processing apparatus.

Further, according to another aspect of the present invention, in the controller, the determination unit may be in a normal state, a state in which the image processing apparatus is likely to be in failure, and the image processing apparatus may be in failure. A configuration may be used in which one of the high states is determined as the state of the image processing apparatus.
With this configuration, the controller determines whether the image processing apparatus is in a normal state, a state in which the image processing apparatus is likely to have failed, or a state in which the image processing apparatus is likely to fail in the future. Judge as. As a result, the controller has one of a normal state of the image processing device, a state in which the image processing device is likely to be in failure, and a state in which the image processing device is likely to fail in the future. This can be notified as the state of the image processing apparatus.

According to another aspect of the present invention, in the controller, the state information includes information indicating one or more physical quantities indicating a state of hardware included in the image processing apparatus, and is based on the one or more physical quantities. The configuration includes an event detection unit that detects an event indicating an abnormality of the image processing device, and the determination unit determines a state of the image processing device based on the event detected by the event detection unit. May be.
With this configuration, the controller detects an event indicating an abnormality of the image processing device based on one or more physical quantities indicating the state of hardware included in the image processing device, and determines the state of the image processing device based on the detected event. to decide. Accordingly, the controller can generate a transmission signal based on the state of the image processing apparatus determined based on the event indicating the abnormality of the image processing apparatus.

According to another aspect of the present invention, in the controller, the state information includes an error code related to control of the image processing apparatus, and the event detection unit is based on an error code related to control of the image processing apparatus. A configuration for detecting an event indicating an abnormality of the image processing apparatus may be used.
With this configuration, the controller detects an event indicating an abnormality of the image processing apparatus based on an error code related to the control of the image processing apparatus. Accordingly, the controller generates a transmission signal based on the state of the image processing apparatus determined based on the event indicating the abnormality of the image processing apparatus based on the error code related to the control of the image processing apparatus as the state of the image processing apparatus. be able to.

According to another aspect of the present invention, in the controller, the determination unit associates information identifying the state of the image processing device with the state of the image processing device determined based on the event, and generates the The configuration may be such that the section includes information for identifying the state of the image processing apparatus associated with the determination section in the transmission signal.
With this configuration, the controller associates information for identifying the state of the image processing apparatus with the state of the image processing apparatus determined based on the event indicating the abnormality of the image processing apparatus, and sets the state of the associated image processing apparatus. Identification information is included in the transmission signal. Thereby, the controller can generate a transmission signal including information for identifying the state of the image processing apparatus.

In another aspect of the present invention, the controller may include a communication control unit that causes the communication unit to transmit the transmission signal generated by the generation unit to the external device.
With this configuration, the controller causes the communication unit to transmit a transmission signal generated based on the determination result of the state of the image processing apparatus to the external device. Thereby, the controller can notify the user of the external device of the state of the image processing apparatus.

According to another aspect of the present invention, in the controller, the communication control unit causes the communication unit to transmit information indicating the one or more physical quantities to the external device in response to a request from the external device. May be used.
With this configuration, the controller causes the external device to transmit information indicating one or more physical quantities indicating the state of the hardware included in the image processing apparatus in the communication unit in response to a request from the external device. Accordingly, the controller can notify the user of one or more physical quantities indicating the state of hardware included in the image processing apparatus.

According to another aspect of the present invention, in the controller, the communication control unit requests the communication unit to change an output state of the output unit included in the image processing device in accordance with the determination result by the determination unit. A configuration in which information is transmitted to the image processing apparatus may be used.
With this configuration, the controller causes the communication unit to transmit information requesting to change the output state of the output unit included in the image processing apparatus to the image processing apparatus in accordance with the determination result of the state of the image processing apparatus. Thereby, the controller can make a user confirm the state of an image processing apparatus with an image processing apparatus.

Another aspect of the present invention is an information processing apparatus that displays a GUI for displaying the one or more physical quantities acquired from the controller described above.
With this configuration, the information processing apparatus displays a GUI for displaying one or more physical quantities indicating the state of hardware included in the image processing apparatus acquired from the controller. Accordingly, the information processing apparatus can facilitate management of the state of the image processing apparatus by providing the user with a GUI that displays one or more physical quantities indicating the state of the hardware included in the image processing apparatus.

In another aspect of the present invention, a robot that performs a predetermined operation, an imaging device that captures a range related to the predetermined operation, an image processing device that processes a captured image captured by the imaging device, and the image processing device And a controller for controlling the robot based on the processing result of the processing, the controller receiving a status information of the image processing device from the image processing device, and the image processing based on the status information. A robot system including a determination unit that determines a state of the apparatus and a generation unit that generates a transmission signal to be transmitted to an external device based on a determination result by the determination unit.
With this configuration, the processing system receives the status information of the image processing apparatus from the image processing apparatus, determines the status of the image processing apparatus based on the status information, and transmits a transmission signal to the external device based on the determination result Is generated. Thereby, the robot system can notify the state of the image processing apparatus.

As described above, the controller and the robot system receive the status information of the image processing apparatus from the image processing apparatus, determine the status of the image processing apparatus based on the status information, and transmit to the external device based on the determination result. A transmission signal is generated. Thereby, the controller and the robot system can notify the state of the image processing apparatus.
Further, the information processing apparatus displays a GUI for displaying one or more physical quantities indicating the state of hardware included in the image processing apparatus acquired from the controller. Accordingly, the information processing apparatus can facilitate management of the state of the image processing apparatus by providing the user with a GUI that displays one or more physical quantities indicating the state of the hardware included in the image processing apparatus.

It is a lineblock diagram showing an example of robot system 1 concerning this embodiment. 2 is a diagram illustrating an example of a hardware configuration of a control device 30. FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus 40. FIG. 3 is a diagram illustrating an example of a functional configuration of a control device 30. FIG. 3 is a diagram illustrating an example of a functional configuration of an image processing device 40. FIG. 4 is a flowchart illustrating an example of a flow of processing performed by a control unit 37 of the control device 30. Correspondence relationship between event indicating abnormality of image processing device 40, condition for detecting event indicating abnormality of image processing device 40, state of image processing device 40, and status code for identifying state of image processing device 40 It is a figure which shows an example. It is a table | surface which shows an example of the rule from which the output control part 474 changes the lighting state of the output part 45. FIG. It is a table | surface which shows an example of the correspondence of the status code which identifies the state of the image processing apparatus 40 determined by the state determination part 373 in step S130, and the message recorded on an error history with a status code. 4 is a diagram illustrating an example of a GUI that displays each detection value included in detection information acquired from the image processing apparatus 40. FIG.

<Embodiment>
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram illustrating an example of a robot system 1 according to the present embodiment. The robot system 1 includes an information processing device 5, an imaging device 10, a robot 20, a control device 30, and an image processing device 40.

  The robot system 1 accepts an operation from the user by the information processing device 5 and causes the robot 20 to perform a predetermined operation based on the accepted operation. For example, as shown in FIG. 1, the predetermined work indicates a work for rearranging the work target M arranged on the upper surface of the work table TB to a predetermined position, but may be another work. The work table TB is, for example, a table or the like, but may be a floor surface, a wall surface, or the like as long as the work object M can be arranged instead. The work target M is an object that can be gripped by the robot 20, and is, for example, a part of an industrial product such as a screw or bolt, but may be another object. In FIG. 1, the work object M is shown as a rectangular parallelepiped object.

  The information processing apparatus 5 is an apparatus capable of installing an application program for controlling the robot 20, such as a notebook PC (Personal Computer), a desktop PC, a tablet PC, or a multi-function mobile phone terminal (smartphone). The apparatus further includes a display. This application program displays a GUI (Graphical User Interface) for creating a control program for controlling the robot 20, and states of the control device 30 and the image processing device 40 (for example, various statuses and the presence / absence of a failure). For example, a GUI having various functions such as a GUI for displaying information is displayed on the display of the information processing apparatus 5.

  The information processing apparatus 5 accepts an operation from the user via the GUI displayed on the display, creates the above-described control program based on the accepted operation, and compiles it. By this compilation, the control program is converted into an object code that can be executed by the control device 30. The information processing device 5 accepts an operation from the user via the GUI displayed on the display, and outputs this object code to the control device 30 based on the accepted operation.

  The information processing device 5 is communicably connected to the control device 30 via a cable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB (Universal Serial Bus), for example. The information processing device 5 and the control device 30 may be configured to be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The imaging device 10 is a camera including, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) that is an imaging device that converts collected light into an electrical signal. The imaging device 10 is connected to the image processing device 40 via a cable so as to be communicable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The imaging device 10 and the image processing device 40 may be configured to be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The imaging device 10 is installed at a position where a range including the work target M can be captured as a captured image. The imaging device 10 is configured to capture a still image as a captured image, but may be configured to capture a moving image as a captured image instead.

  The robot 20 acquires a control signal based on the position and orientation of the work target M from the control device 30, and performs a predetermined work based on the acquired control signal. The robot 20 is, for example, a single-arm robot including an end effector END having a claw portion that can grip an object (in this example, the work target M), a manipulator MNP, and a plurality of actuators (not shown). The single-arm robot refers to a robot having one arm constituted by the end effector END and the manipulator MNP (or only the manipulator MNP).

  The robot 20 may be a SCARA robot (horizontal articulated robot), a double-arm robot, or the like instead of the single-arm robot. A SCARA robot is a robot in which the manipulator moves only in the horizontal direction and only the slide shaft at the tip of the manipulator moves up and down. The dual-arm robot refers to a robot having two arms each constituted by an end effector END and a manipulator MNP (or only the manipulator MNP).

  The arm of the robot 20 is a 6-axis vertical articulated type, and the support base, the manipulator MNP, and the end effector END can perform an operation with six degrees of freedom by an operation in which the actuator is linked. Note that the arm of the robot 20 may operate with 5 degrees of freedom (5 axes) or less, or may operate with 7 degrees of freedom (7 axes) or more.

  The robot 20 is communicably connected to the control device 30 by a cable, for example. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The robot 20 and the control device 30 may be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). The robot 20 is configured to be connected to the control device 30 installed outside the robot 20 as shown in FIG. 1. However, instead of this configuration, the control device 30 is built in the robot 20. It may be.

  The control device 30 acquires the object code from the information processing device 5, and controls the robot 20 to perform a predetermined operation based on the acquired object code. More specifically, in this example, the control device 30 acquires and acquires a captured image in which the range including the work target M is captured by the imaging device 10 from the image processing device 40 based on the object code. A request is made to perform image processing for calculating the position and orientation of the work object M based on the captured image.

  Then, after the above-described image processing by the image processing device 40 is completed, the control device 30 acquires information indicating the position and orientation of the work target M from the image processing device 40. The control device 30 generates a control signal based on the acquired position and orientation of the work target M, and outputs the generated control signal to the robot 20 so that the robot 20 performs a predetermined work.

  In response to a request from the control device 30 based on the object code, the image processing device 40 acquires a captured image in which a range including the work target M is captured from the imaging device 10. Then, after acquiring the captured image from the imaging device 10, the image processing device 40 performs image processing for calculating the position and orientation of the work target M based on the acquired captured image. The image processing device 40 outputs information indicating the position and orientation of the work target M obtained as a result of the image processing to the control device 30.

  Note that the image processing device 40 periodically acquires captured images from the imaging device 10 instead of a configuration that performs these processes in response to a request from the control device 30 based on the object code, and acquires the captured images. The position and orientation of the work target M may be calculated by performing the above-described image processing on the acquired captured image each time. In this case, every time the image processing apparatus 40 acquires a request for acquiring information indicating the position and orientation of the work target M from the control device 30, information indicating the position and orientation of the work target M calculated at that time point. Is output to the control device 30.

  Hereinafter, the abnormal state notification function of the image processing apparatus included in the control device 30 in the robot system 1 configured as described above will be described. The abnormal state notification function is, for example, a part of one or more hardware functional units provided in the image processing device 40 or a part of software control processing, such as an excessive increase in the temperature of the CPU provided in the image processing device 40 All of them are functions for detecting an event indicating a state of high possibility of failure or a state of high possibility of failure in the future and performing notification based on the detected event.

  In the following, for convenience of explanation, a part or all of one or more hardware functional units included in the image processing apparatus 40 or a control process by software is likely to have failed or is likely to fail in the future. The event indicating the state is referred to as an event indicating an abnormality of the image processing apparatus 40. In this example, the state where the possibility of failure is high is a state in which the probability of failure within a predetermined time exceeds a predetermined value when the use state of the image processing apparatus 40 is maintained and used in the current state. However, other states may be indicated.

  Next, the hardware configuration of the control device 30 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the control device 30. The control device 30 includes, for example, a CPU (Central Processing Unit) 31, a storage unit 32, an input reception unit 33, a communication unit 34, and an output unit 35, and the information processing device 5 and the image via the communication unit 34. It communicates with the processing device 40 and other devices. These components are connected to each other via a bus Bus so that they can communicate with each other. The CPU 31 executes various programs stored in the storage unit 32.

The storage unit 32 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a RAM (Random Access Memory), and the like. Various information, images, programs and the like processed by the control device 30 are stored.
Note that the storage unit 32 may be an external storage device connected by a digital input / output port such as a USB instead of the one built in the control device 30.

The input receiving unit 33 may be, for example, a keyboard, a mouse, a touch pad, or other input device. Moreover, the input reception part 43 may be comprised integrally with a display as a touchscreen.
The communication unit 34 includes, for example, a plurality of Ethernet (registered trademark) ports and a plurality of digital input / output ports such as USB.
The output unit 45 may have, for example, a configuration including a liquid crystal display panel or an organic EL (ElectroLuminescence) display panel, or a configuration including a speaker that outputs sound.

  Next, the hardware configuration of the image processing apparatus 40 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a hardware configuration of the image processing apparatus 40. The image processing device 40 includes, for example, a CPU 41, a storage unit 42, an input reception unit 43, a communication unit 44, an output unit 45, and a detection unit 46, and the control device 30 and other devices via the communication unit 44. It communicates with the image processing apparatus 40 and other apparatuses. These components are connected to each other via a bus Bus so that they can communicate with each other. The CPU 41 executes various programs stored in the storage unit 42.

The storage unit 42 includes a memory 421 and a storage 422.
The memory 421 includes, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), and the like, and is a storage device that can be directly accessed mainly by a processor such as the CPU 41 among the functional units included in the image processing apparatus 40.

The storage 422 includes, for example, an HDD, an SSD, an EEPROM, a ROM, a RAM, and the like, and stores various information, images, programs, and the like that are processed by the image processing apparatus 40.
Each of the memory 421 and the storage 422 may be an external storage device connected by a digital input / output port such as a USB instead of the one built in the image processing apparatus 40.

The input receiving unit 43 is one or more buttons that can input a plurality of requests to the image processing device 40 by pressing or long pressing, for example. Instead, a keyboard, a mouse, a touch pad, Other input devices may be used. In that case, the input reception unit 43 may be configured integrally with the display unit as a touch panel. In this example, the input receiving unit 43 will be described as one button.
The communication unit 44 includes an external communication unit 441, a first imaging communication unit 442, and a second imaging communication unit 443.

The external communication unit 441 includes, for example, a plurality of Ethernet (registered trademark) ports and a plurality of digital input / output ports such as USB. In this example, the external communication unit 441 will be described as having a configuration including two Ethernet (registered trademark) ports.
The first imaging communication unit 442 includes a plurality of Ethernet (registered trademark) ports, a plurality of digital input / output ports such as a USB, and the like, and these ports are dedicated ports for communicating with the imaging device 10. is there. The first imaging communication unit 442 may be a dedicated port for communicating with other devices such as a sound acquisition device for acquiring sound.

The second imaging communication unit 443 includes a plurality of Ethernet (registered trademark) ports, a plurality of digital input / output ports such as USB, and the like, and these ports are dedicated ports for communicating with the imaging device 10. is there. Note that the second imaging communication unit 443 may be a dedicated port for communicating with other devices such as a sound acquisition device for acquiring sound.
The first imaging communication unit 442 and the second imaging communication unit 443 have different port communication standards. In this example, the first imaging communication unit 442 is configured to include four Ethernet (registered trademark) ports, and the second imaging communication unit 443 is configured to include four USB ports.

  The output unit 45 includes a plurality of LEDs, and notifies various types of information according to the lighting states of these LEDs (eg, extinguishing, lighting, blinking, etc.). In this example, the output unit 45 is described as including one green LED and one red LED. However, the colors of these LEDs may be other colors or the same color. Good. The output unit 45 may be configured to include a liquid crystal display panel or an organic EL display panel, for example, instead of a configuration including these LEDs, and includes a speaker that outputs sound. Also good.

  The detection unit 46 is a plurality of sensors for detecting a physical quantity that serves as an index indicating that an event indicating an abnormality of each of a plurality of pieces of hardware included in the image processing apparatus 40 has occurred. The event indicating a hardware abnormality refers to an event in which the image processing device 40 is replaced with hardware in the definition of the event indicating the abnormality of the image processing device 40 described above. In this example, the hardware included in the image processing device 40 includes a cooling fan for the CPU 41 (not shown), a system fan for cooling the inside of the housing of the image processing device 40 (not shown), the CPU 41, and an image processing device (not shown). 40 shows a battery for BIOS (Basic Input Output System) backup built in 40, but it may be a combination of some of these hardware, or a combination of these hardware including other hardware. It may be one or more other hardware.

  In this example, the physical quantity serving as an index indicating that an event indicating an abnormality of the hardware included in the image processing apparatus 40 has occurred is the number of rotations of the cooling fan of the CPU 41 and the inside of the casing of the image processing apparatus 40. The rotational speed of the system fan that cools the CPU, the temperature of the CPU 41, and the voltage value of the BIOS backup battery built in the image processing device 40, but instead of these, a combination of these physical quantities may be used. These physical quantities may include a combination including other physical quantities, or may be one or more other physical quantities. Note that the physical quantity serving as an index indicating that an event indicating an abnormality of each of the plurality of hardware included in the image processing apparatus 40 is an example of one or more physical quantities indicating the state of the hardware included in the own apparatus.

In the present embodiment, the detection unit 46 includes four sensors, for example, a first detection unit 461, a second detection unit 462, a third detection unit 463, and a fourth detection unit 464.
The first detection unit 461 is, for example, a rotation speed sensor that detects the rotation speed of the cooling fan of the CPU 41.
The second detection unit 462 is, for example, a rotation speed sensor that detects the rotation speed of a system fan that cools the inside of the housing of the image processing apparatus 40.
The third detection unit 463 is, for example, a temperature sensor that detects the temperature of the CPU 41.
The fourth detection unit 464 is, for example, a voltage sensor that detects a voltage value of a BIOS backup battery built in the image processing apparatus 40.

Next, the functional configuration of the control device 30 will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a functional configuration of the control device 30. The control device 30 includes a storage unit 32, a communication unit 34, an output unit 35, and a control unit 37.
The control unit 37 includes a detection information acquisition unit 371, an abnormal event detection unit 372, a state determination unit 373, an output control unit 374, a communication control unit 375, and a robot control unit 376. A part or all of these functional units included in the control unit 37 is realized by the CPU 31 executing a program stored in the storage unit 32, for example. Some of these functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

  The detection information acquisition unit 371 acquires detection information from the image processing device 40 via the communication unit 34 periodically (for example, every 1 minute). In this example, the detection information includes physical quantities detected by each of the first detection unit 461 to the fourth detection unit 464 included in the detection unit 46 of the image processing apparatus 40 and a memory 421 included in the storage unit 42 of the image processing apparatus 40. And information indicating the free capacity of each storage area of the storage 422 and information indicating an error code related to the processing of the image processing apparatus 40. The detection information acquisition unit 371 may be configured to constantly acquire detection information instead of the configuration to periodically acquire detection information from the image processing device 40, and may be configured to acquire at a predetermined time. Also good. The detection information acquisition unit 371 is an example of a receiving unit. The detection information is an example of state information.

  The abnormal event detection unit 372 detects an event indicating an abnormality of the image processing device 40 acquired from the image processing device 40 by the detection information acquisition unit 371. More specifically, the abnormal event detection unit 372 detects an event indicating a hardware abnormality of the image processing apparatus 40 based on the detection information acquired by the detection information acquisition unit 371. Further, the abnormal event detection unit 372 detects an event indicating an abnormality related to the control of the image processing device 40 (control processing by the above-described software) based on the error code related to the processing of the image processing device 40. The abnormal event detection unit 372 is an example of an event detection unit.

When the abnormal event detection unit 372 detects an event indicating an abnormality of the image processing device 40, the state determination unit 373 determines whether the image processing device 40 described below is based on the detected event indicating the abnormality of the image processing device 40. Determine the state. The state determination unit 373 is an example of a determination unit.
The output control unit 374 causes the output unit 35 to display information indicating the determination result of the state of the image processing apparatus 40 by the state determination unit 373. For example, when the output unit 35 is a speaker, the output control unit 374 causes the output unit 35 to output sound representing information indicating the determination result of the state of the image processing apparatus 40 by the state determination unit 373.

  When the abnormal event detection unit 372 detects an event indicating an abnormality of the image processing device 40 or when a request from the information processing device 5 is acquired, the communication control unit 375 determines whether the image processing determined by the state determination unit 373 is performed. A transmission signal including information indicating the state of the device 40 is generated, and the generated transmission signal is output to the information processing device 5 via the communication unit 34. In the following, for convenience of explanation, a communication control unit 375 generates a transmission signal including information P, and outputs the generated transmission signal to an external device (for example, the information processing apparatus 5) via the communication unit 34. Is simply referred to as the communication control unit 375 causing the communication unit 34 to output the information P to an external device.

  Further, in this example, the information indicating the state of the image processing device 40 is a numerical value that identifies the state of the image processing device 40. However, other information that identifies the state of the image processing device 40, such as characters and symbols, is used. It may be information. Hereinafter, for convenience of explanation, this numerical value will be referred to as a status code. The status code is an example of information for identifying the status of the image processing apparatus. In addition, the communication control unit 375 requests the communication unit 34 to change the output state of the output unit 45 included in the image processing apparatus 40 to the output state according to the determination result determined by the state determination unit 373. Output to the device 40. The communication control unit 375 is an example of each of the generation unit and the communication control unit.

  The robot control unit 376 acquires an object code from the information processing device 5 via the communication unit 34, and causes the robot 20 to perform a predetermined operation based on the acquired object code. When this predetermined work is performed, the robot control unit 376 acquires a captured image in which the range including the work target M is captured by the imaging device 10 from the image processing device 40 based on the object code. A request is made to perform image processing for calculating the position and orientation of the work object M based on the captured image.

  The robot control unit 376 acquires information indicating the position and orientation of the work target M from the image processing device 40 after the above-described image processing by the image processing device 40 is completed. The robot control unit 376 generates a control signal based on the acquired position and orientation of the work target M, and outputs the generated control signal to the robot 20 to cause the robot 20 to perform a predetermined work.

  Next, the functional configuration of the image processing apparatus 40 will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of a functional configuration of the image processing apparatus 40. The image processing apparatus 40 includes a storage unit 42, a communication unit 44, an output unit 45, a detection unit 46, and a control unit 47. In this example, it is assumed that the imaging device 10 is connected to the first imaging communication unit 442 and the second imaging communication unit 443 is not connected to the imaging device.

  The control unit 47 controls the entire image processing apparatus 40. The control unit 47 includes a detection information acquisition unit 471, an abnormal event detection unit 472, a state determination unit 473, an output control unit 474, a communication control unit 475, an imaging control unit 476, an image acquisition unit 477, an image A processing unit 478 is provided. A part or all of these functional units included in the control unit 47 is realized by the CPU 41 executing a program stored in the storage unit 42, for example. Some of these functional units may be hardware functional units such as LSI and ASIC.

  The detection information acquisition unit 471 periodically acquires a physical quantity detected by each of the first detection unit 461 to the fourth detection unit 464 included in the detection unit 46 (for example, every 1 minute). In addition, the detection information acquisition unit 471 periodically acquires information indicating the free capacity of the storage area from each of the memory 421 and the storage 422 included in the storage unit 42 (for example, every 1 minute). Hereinafter, for convenience of explanation, a plurality of physical quantities acquired from the detection unit 46 and information indicating two free capacities acquired from the storage unit 42 will be collectively referred to as detection information. The detection information acquisition unit 471 may be configured to constantly acquire detection information instead of the configuration to periodically acquire detection information, or may be configured to acquire at a predetermined time.

The output control unit 474 changes the output state of the output unit 45 (in this example, the lighting state) in response to a request acquired from the control device 30 via the external communication unit 441.
The communication control unit 475 causes the external communication unit 441 to output information indicating the position and orientation of the work target M calculated by the image processing unit 478 to the control device 30.

  The imaging control unit 476 causes the imaging device 10 to image a range including the work target M via the first imaging communication unit 442. In addition, when any imaging device Z is connected to the second imaging communication unit 443, the imaging control unit 476 captures an image capturing range of the imaging device Z in the imaging device Z via the second imaging communication unit 443. Let

The image acquisition unit 477 acquires a captured image from the imaging device 10 via the first imaging communication unit 442. Further, the image acquisition unit 477 acquires a captured image from the imaging device Z via the second imaging communication unit 443 when the imaging device Z is connected to the second imaging communication unit 443.
The image processing unit 478 performs image processing for calculating the position and orientation of the work target M based on the captured image acquired by the image acquisition unit 477.

  Hereinafter, processing performed by the control unit 37 of the control device 30 will be described with reference to FIG. FIG. 6 is a flowchart illustrating an example of a flow of processing performed by the control unit 37 of the control device 30. In the present embodiment, the control unit 37 performs the processing flow illustrated in FIG. 6 periodically (for example, every 1 minute) while the image processing apparatus 40 connected to the control apparatus 30 is powered on. However, it may be configured to be performed at a predetermined time or may be performed constantly. The control device 30 may be configured to perform the processing flow illustrated in FIG. 6 in response to a request from the information processing device 5.

  First, the detection information acquisition unit 371 outputs a detection information acquisition request to the image processing apparatus 40 via the communication unit 34. And the detection information acquisition part 371 acquires detection information from the image processing apparatus 40 as a response with respect to the request | requirement via the communication part 34 (step S110). Next, the abnormal event detection unit 372 detects an event indicating an abnormality of the image processing apparatus 40 based on the detection information acquired by the detection information acquisition unit 471 in step S110.

  In this example, it is assumed that the events indicating abnormality of the image processing apparatus 40 are the 13 events from A) to M) described below. The event indicating the abnormality of the image processing apparatus 40 may be a part of the 13 events up to A) to M) shown below, and in addition to A) to M) shown below. Other events may be included, and some of the events including other events and A) to M) described below may be included. In addition, A) to M) shown below, along with an event indicating an abnormality of the image processing apparatus 40, shows the conditions under which each is detected.

A) The temperature of the CPU 41 included in the detection information is within a predetermined range X1 (for example, 90 ° C. or higher and lower than 100 ° C.) B) The temperature of the CPU 41 included in the detection information is a predetermined threshold Y1 (for example, 100 ° C.) or higher C) Detection information The rotation speed of the cooling fan of the CPU 41 included in the CPU 41 is within a predetermined range X2 (for example, less than 1000 RPM instead of 0 RPM (D)) The rotation speed of the cooling fan of the CPU 41 included in the detection information is a predetermined threshold Y2 (for example, 0 RPM) or less E) The number of revolutions included in the detection information and the number of revolutions of the system fan that cools the inside of the housing of the image processing apparatus 40 is within a predetermined range X3 (for example, less than 1000 RPM instead of 0 RPM (Revolution Per Minutes)). F) The number of rotations included in the detection information and the number of rotations of the system fan that cools the inside of the housing of the image processing apparatus 40 Threshold value Y3 (for example, 0 RPM or less G) The voltage value included in the detection information and the voltage value of the BIOS backup battery built in the image processing apparatus 40 is within a predetermined range X4 (for example, 2 V or more and less than 2.5 V). H) The voltage value included in the detection information and the voltage value of the BIOS backup battery built in the image processing apparatus 40 is less than a predetermined threshold Y4 (for example, less than 2V) I) The error code included in the detection information is commanded J) The code indicating the error is included. J) The free area of the memory 421 included in the detection information is less than a predetermined threshold Y5 (for example, 10 MByte). K) The free area of the storage 422 included in the detection information is the predetermined threshold Y6 (for example, 10 MByte). L) The error code included in the detection information includes a code indicating a detection information acquisition error by the detection information acquisition unit 471. M) The error code included in the detection information includes a code indicating a control error of the output unit 45 by the output control unit 474.

  In the following, for convenience of explanation, it is assumed that the event A) among the events A) to M) shown above is detected by the abnormal event detection unit 372 in step S120.

  Next, the state determination unit 373 performs image processing based on the event indicating the abnormality of the image processing apparatus 40 detected in step S130 by the abnormal event detection unit 372 (in this example, the event A described above). The state of the device 40 is determined (step S130). Here, processing for determining the state of the image processing apparatus 40 by the state determination unit 373 will be described. The state determination unit 373 determines the state of the image processing device 40 from the correspondence as shown in FIG. 7 based on the event indicating the abnormality of the image processing device 40 detected by the abnormal event detection unit 372 in step S130. . FIG. 7 shows an event indicating an abnormality of the image processing device 40, a condition for detecting an event indicating an abnormality of the image processing device 40, a state of the image processing device 40, and a status code for identifying the state of the image processing device 40 It is a figure which shows an example of a corresponding relationship.

  For example, the state determination unit 373 sets the state of the image processing device 40 to the normal state (when the abnormal event detection unit 372 detects no abnormality indicating the abnormality of the image processing device 40 based on the correspondence shown in FIG. It is determined that the status code is 0). The normal state is an example of a normal state. When the abnormal event detection unit 372 detects a CPU temperature abnormality (the detected condition is 90 ° C. or higher and lower than 100 ° C.) as an event indicating an abnormality of the image processing device 40, the state determination unit 373 Is determined to be a warning occurrence state. The warning occurrence state indicates a state in which one or more hardware functional units provided in the image processing apparatus 40 or a part or all of the control processing by software is likely to fail.

  In addition, the state determination unit 373 determines the state of the image processing device 40 when the abnormal event detection unit 372 detects a CPU temperature abnormality (the detected condition is 100 ° C. or higher) as an event indicating an abnormality of the image processing device 40. It is determined that an error has occurred. The error occurrence state indicates a state in which there is a high possibility that a part or all of the functional unit by one or more hardware included in the image processing apparatus 40 or the control processing by software is out of order.

  The correspondence relationship shown in FIG. 7 is merely an example, and an event indicating an abnormality of the image processing device 40 may be associated with the state or status code of another image processing device 40. Further, the state of the image processing apparatus 40 may be any one of these three instead of any of the normal state, the warning occurrence state, and the error occurrence state, and is different from these three. It may be one of one or more other states, or any of four or more including these three other states.

  In this example, the event indicating the abnormality of the image processing apparatus 40 detected by the abnormal event detection unit 372 in step S130 is associated with the event A (ie, status code 1 shown in FIG. 7). In step S30, the state determination unit 373 determines that the state of the image processing device 40 is a warning occurrence state.

  Next, the communication control unit 375 determines the lighting state of the output unit 45 included in the image processing apparatus 40 as the state of the image processing apparatus 40 detected by the state determination unit 373 in step S130 (determination determined by the state determination unit 373). A request for changing the lighting state according to the result is output to the image processing apparatus 40 by the communication unit 34 (step S130). When the output control unit 474 included in the image processing apparatus 40 acquires this request, the output control unit 474 changes the lighting state of the output unit 45 according to the acquired request. Here, a process of changing the lighting state of the output unit 45 by the output control unit 474 will be described. Based on the correspondence shown in FIG. 8, the output control unit 474 causes the output unit 45 to change the lighting state according to the state of the image processing device 40 detected by the state determination unit 373 in step S <b> 130. FIG. 8 is a table showing an example of a rule that causes the output control unit 474 to change the lighting state of the output unit 45.

  When the state of the image processing apparatus 40 detected by the state determination unit 373 is a warning occurrence state, the output control unit 474 causes the output unit 45 to change the lighting state of the output unit 45 to blinking. On the other hand, when the state of the image processing apparatus 40 detected by the state determination unit 373 is an error occurrence state, the output control unit 474 causes the output unit 45 to change the lighting state of the output unit 45 to lighting. In this example, since the state of the image processing apparatus 40 detected by the state determination unit 373 in step S130 is a warning occurrence state, the output control unit 474 causes the output unit 45 to change the lighting state of the output unit 45 to blinking. .

  In this example, when the output control unit 474 causes the output unit 45 to change the lighting state of the red LED included in the output unit 45 to blinking or lighting, the state determination unit 373 determines that it is in a normal state. (That is, when the state of the image processing apparatus 40 is restored from a warning occurrence state or an error occurrence state to a normal state), the output unit 45 is configured to change the lighting state of the red LED included in the output unit 45 to off. However, it may be configured to change to a lighting state other than blinking, lighting, and extinguishing.

Further, the output control unit 474 is configured so that the state determination unit 373 determines, for example, that the warning occurrence state related to the CPU 41 and the error occurrence state related to the system fan are occurring at the same time. When a plurality of events indicating 40 abnormalities are detected and the state of the image processing apparatus 40 is determined for each of them, for example, the lighting state of the output unit 45 is changed to the output unit 45 according to the following rules R1) and R2). Change it.
R1) When the error occurrence state and the warning occurrence state occur simultaneously, change the lighting state of the output unit 45 to lighting. R2) When the error occurrence state does not occur and the warning occurrence state occurs, the output unit Change the lighting state of 45 to blinking

  Next, the communication control unit 375 causes the external communication unit 441 to output the information indicating the status code for identifying the status of the image processing device 40 determined by the status determination unit 473 in step S130 to the information processing device 5 (step S130). S150). Thus, the control unit 37 repeats the processing from step S110 to step S150, thereby detecting an event indicating abnormality of the image processing device 40, and according to the detected event indicating abnormality of the image processing device 40. The lighting state of the output unit 45 is changed. Accordingly, the control device 30 can notify the user of the state of the image processing device 40 by the lighting state of the output unit of the image processing device 40. Further, the control unit 37 can notify the information processing device 5 of the state of the image processing device 40 by repeating the processing from step S110 to step S150.

  Here, processing performed by the control device 30 and the information processing device 5 after acquiring information indicating the status code output from the image processing device 40 in step S150 will be described. The control device 30 records the status code for identifying the status of the image processing device 40 determined by the status determination unit 373 in step S130 together with the message associated with the status code shown in FIG. 9 in the error history. FIG. 9 is a table showing an example of a correspondence relationship between a status code for identifying the status of the image processing apparatus 40 determined by the status determination unit 373 in step S130 and a message recorded in the error history together with the status code.

  As described above, the control device 30 records, as an error history, which hardware or control of the image processing device 40 causes an event indicating an abnormality of the image processing device 40, and allows the user to perform maintenance or maintenance of the image processing device 40. It is possible to reduce the labor involved in repairs. Note that the status code illustrated in FIG. 9 is merely an example, and other numerical values, characters, and the like may be used.

  In addition, the control device 30 includes a display unit W that is different from the output unit 35, such as a 7-segment LED, and causes the output control unit 374 to display a number representing a status code on the display unit W, so that the image is displayed to the user. The structure which notifies that the event which shows abnormality of the processing apparatus 40 occurred may be sufficient. In this case, the control device 30 can easily notify the user of which hardware or control of the image processing device 40 causes an event indicating an abnormality of the image processing device 40.

  Further, the communication control unit 375 of the control device 30 controls the communication unit 34 so as to output information indicating the status code to the information processing device 5. The information processing device 5 acquires information indicating a status code from the control device 30, and displays information indicating that an event indicating an abnormality has occurred in the image processing device 40 based on the acquired status code (or display Displayed on the GUI). Thereby, the information processing apparatus 5 notifies the user of which hardware or control of the image processing apparatus 40 causes an event indicating an abnormality of the image processing apparatus 40 to the user via the GUI. As a result, it is possible to reduce the time and effort required for the user to directly operate the control device 30 and the image processing device 40.

  Hereinafter, a process performed between the information processing apparatus 5 and the control apparatus 30 after the image processing apparatus 40 is connected to the robot system 1 (in this example, the control apparatus 30) will be described. After the image processing apparatus 40 is connected to the control apparatus 30 and recognizes the image processing apparatus 40, the information processing apparatus 5 periodically acquires detection information acquired from the image processing apparatus 40 by the control apparatus 30 (for example, Request every 1 second). When acquiring this request, the control device 30 outputs detection information acquired by the control device 30 to the information processing device 5 at the time when this request is acquired.

  The information processing device 5 acquires detection information from the control device 30. The information processing apparatus 5 receives an operation from the user via the GUI displayed on the display, and displays a GUI for displaying each detection value included in the acquired detection information based on the received operation on the display. . FIG. 10 is a diagram illustrating an example of a GUI that displays each detection value included in the detection information acquired from the image processing apparatus 40.

  In this example, the GUI includes a free capacity of the memory 421 (shown as aMB of the total capacity bMB in FIG. 10) and a free capacity of the storage 422 (shown in FIG. Of the capacity dMB), the rotation speed of the cooling fan of the CPU 41 (shown as eRPM in FIG. 10), the temperature of the CPU 41 (shown as f.degree. C. in FIG. 10), and the image processing apparatus. The rotational speed of the system fan that cools the inside of the housing 40 (shown as gRPM in FIG. 10) and the voltage value of the BIOS backup battery built in the image processing apparatus 40 (shown as hV in FIG. 10) ) Is displayed. Thereby, the user can confirm whether or not an event indicating an abnormality of the image processing apparatus 40 has occurred by displaying the GUI shown in FIG. 10 on the display of the information processing apparatus 5.

  Note that the processing performed between the information processing device 5 and the control device 30 may be performed between the control device 30 and the image processing device 40, or performed between the information processing device 5 and the image processing device 40. Alternatively, the processing may be performed between the information processing device 5 and the control device 30, and the image processing device 40.

The image processing device 40 is merely an example of a processing device, and the abnormal state notification function described in the present embodiment may be included in, for example, a voice processing device that processes voice data. An input processing device (for example, a teaching pendant) that processes the operation and moves the robot 20 according to the input operation may have, or a data processing device that processes some other data.
The control device 30 described above is an example of a controller.

  As described above, the control device 30 included in the robot system 1 in the present embodiment acquires (receives) detection information of the image processing device 40 from the image processing device 40 that processes the captured image, and based on the detection information. Then, the state of the image processing device 40 is determined, and a transmission signal to be transmitted to an external device (in this example, the information processing device 5) is generated based on the determination result. As a result, the control device 30 can notify the state of the image processing device.

  In addition, the control device 30 includes a state in which the state of the image processing device 40 is normal, a state in which the image processing device 40 is likely to be broken, and a state in which the image processing device 40 is likely to break down. One of them is determined as the state of the image processing apparatus 40. Thereby, the control device 30 has a normal state of the image processing device 40, a state in which the image processing device 40 is likely to be in failure, and a state in which the image processing device 40 is likely to fail in the future. Can be notified as the state of the image processing apparatus 40.

  Further, the control device 30 detects an event indicating an abnormality of the image processing device 40 based on one or more physical quantities indicating the state of hardware included in the image processing device 40, and the image processing device 40 based on the detected event. Determine the state. Thereby, the control apparatus 30 can generate | occur | produce the transmission signal based on the state of the image processing apparatus 40 judged based on the event which shows abnormality of the image processing apparatus 40. FIG.

  Further, the control device 30 detects an event indicating an abnormality of the image processing device 40 based on an error code related to the control of the image processing device 40. Thereby, the control device 30 changes the state of the image processing device 40 to the state of the image processing device 40 determined based on the event indicating the abnormality of the image processing device 40 based on the error code related to the control of the image processing device 40. Based transmission signals can be generated.

  Further, the control device 30 associates information for identifying the state of the image processing device 40 with the state of the image processing device 40 determined based on the event indicating the abnormality of the image processing device 40, and associates the image processing device 40 with the associated information. Information for identifying the 40 states is included in the transmission signal. Thereby, the control device 30 can generate a transmission signal including information for identifying the state of the image processing device 40.

  In addition, the control device 30 causes the communication unit 34 to transmit a transmission signal generated based on the determination result of the state of the image processing device 40 to the external device. Thereby, the control device 30 can notify the user of the external device of the state of the image processing device 40.

  Also, the control device 30 causes the communication unit 34 to transmit information indicating one or more physical quantities indicating the state of hardware included in the image processing device 40 to the external device in response to a request from the external device. Accordingly, the control device 30 can notify the user of one or more physical quantities indicating the state of the hardware included in the image processing device 40.

  Further, the control device 30 causes the communication unit 34 to transmit information requesting to change the lighting state of the output unit 45 included in the image processing device 40 to the image processing device 40 according to the determination result of the state of the image processing device 40. . Thereby, the control device 30 can make the user confirm the state of the image processing device 40 by the image processing device 40.

  Further, the information processing apparatus 5 displays a GUI for displaying one or more physical quantities indicating the state of hardware included in the image processing apparatus 40 acquired from the control apparatus 30. Accordingly, the information processing apparatus 5 facilitates management of the state of the image processing apparatus 40 by providing the user with a GUI that displays one or more physical quantities indicating the state of the hardware included in the image processing apparatus 40. Can do.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and changes, substitutions, deletions, and the like are possible without departing from the gist of the present invention. May be.

  In addition, a program for realizing the functions of arbitrary components in the devices described above (for example, the information processing device 5, the control device 30, and the image processing device 40 of the robot system 1) is recorded on a computer-readable recording medium. The program may be recorded, and the program may be read into a computer system and executed. Here, the “computer system” includes hardware such as an OS (Operating System) and peripheral devices. “Computer-readable recording medium” means a portable disk such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD (Compact Disk) -ROM, or a hard disk built in a computer system. Refers to the device. Further, the “computer-readable recording medium” means a volatile memory (RAM: Random Access) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Memory that holds a program for a certain period of time, such as Memory).

In addition, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above. Further, the program may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

DESCRIPTION OF SYMBOLS 1 Robot system, 5 Information processing apparatus, 10 Imaging device, 20 Robot, 30 Control apparatus, 40 Image processing apparatus, 41 CPU, 42 Storage part, 43 Input reception part, 44 Communication part, 45 Output part, 46 Detection part, 47 Control unit, 421 memory, 422 storage, 441 external communication unit, 442 first imaging communication unit, 443 second imaging communication unit, 461 first detection unit, 462 second detection unit, 463 third detection unit, 464 fourth detection Unit, 471 detection information acquisition unit, 472 abnormal event detection unit, 473 state determination unit, 474 output control unit, 475 communication control unit, 476 imaging control unit, 477 image acquisition unit, 478 image processing unit

Claims (10)

A receiving unit that receives status information of the image processing device from an image processing device that processes the captured image;
A determination unit for determining a state of the image processing device based on the state information;
A generation unit that generates a transmission signal to be transmitted to an external device based on a determination result by the determination unit;
controller. The controller of claim 1,
The determination unit determines whether the image processing apparatus is in a normal state, a state in which the image processing apparatus is likely to be faulty, or a state in which the image processing apparatus is likely to fail in the future. Judge as the state,
controller. The controller according to claim 1 or 2,
The state information includes information indicating one or more physical quantities indicating a state of hardware included in the image processing apparatus,
An event detection unit that detects an event indicating an abnormality of the image processing device based on the one or more physical quantities;
The determination unit determines a state of the image processing apparatus based on the event detected by the event detection unit;
controller. The controller according to claim 3,
The status information includes an error code related to control of the image processing apparatus,
The event detection unit detects the event indicating an abnormality of the image processing device based on an error code related to control of the image processing device;
controller. The controller according to claim 4, wherein
The determination unit associates information identifying the state of the image processing device with the state of the image processing device determined based on the event,
The generation unit includes information for identifying a state of the image processing apparatus associated with the determination unit in the transmission signal.
controller. A controller according to any one of claims 1 to 5,
A communication control unit that causes the communication unit to transmit the transmission signal generated by the generation unit to the external device;
A controller with The controller of claim 6 quoting claim 3.
The communication control unit causes the communication unit to transmit information indicating the one or more physical quantities to the external device in response to a request from the external device.
controller. The controller according to claim 6 or 7, wherein
The communication control unit causes the communication unit to transmit information requesting a change in the output state of the output unit included in the image processing device to the image processing device according to the determination result by the determination unit.
A controller with Displaying a GUI for displaying the one or more physical quantities obtained from the controller of claim 8 quoting claim 3;
Information processing device. A robot that performs predetermined tasks;
An imaging device for imaging a range related to the predetermined operation;
An image processing apparatus that processes a captured image captured by the imaging apparatus;
A controller for controlling the robot based on a result processed by the image processing device,
The controller is
A receiving unit for receiving status information of the image processing device from the image processing device;
A determination unit for determining a state of the image processing device based on the state information;
A generation unit that generates a transmission signal to be transmitted to an external device based on a determination result by the determination unit;
Robot system.

Images