TW201617789A - Control apparatus, system and method - Google Patents

Abstract

A control apparatus includes a detecting module that captures a dynamic image as a target image and displays the target image on a webpage, a processing module that receives and converts the target image into a data instruction, and a transmission module that receives and transmits the data instruction via a network to a machine. Therefore, the control apparatus realizes the control technique for a user to control remotely, greatly improves the convenience, timeliness and accuracy for remote control. The present invention further provides a control system and a control method.

Description

Control device, system and method

The present invention relates to a control system, and more particularly to a control device, system and method that can be used to control the movement of mechanical equipment.

A "manipulator" or "robot" is an automated machine that can be used to manipulate, manipulate, or manipulate tools by mimicking human or physical movements. Taking a robot as an example, the robot performs control commands of fixed or non-fixed programs according to parameters such as coordinates and speed to perform various required actions. Therefore, before the robot executes the control command, it is necessary to perform positioning control of parameters such as coordinates and speed.

As shown in Fig. 1, the positioning control of the robot is mainly for the operator 10 to adjust the position of the manipulator, set and record coordinate parameters by manually adjusting the advance and retreat of the X and Y axes through the operation interface of the control machine 1 in the manual mode. Check the position of the robot, compare the coordinates of the coordinates with the position of the robot to obtain reasonable coordinate parameters.

However, in this manual mode, even a skilled operator needs to operate a physical device (for example, button 11, keyboard or touch screen, etc.) and cumbersome by using a user-friendly operation interface. Step (such as pressing the button 11 for fine adjustment) to perform input and adjustment of the positioning operation Schools, and the operation of buttons by different operators may also cause health problems.

Furthermore, the operator must be at the site where the robot is located to be able to perform positioning control. If the operator is unable to visit the site for any reason, it may cause delay in subsequent work.

Recently, the somatosensory control technology that adopts the separation mode between the client and the server is published. However, the data transmission efficiency between the client and the server side is low, and the data packet is easily lost, resulting in inaccurate data.

In addition, known somatosensory control techniques require additional sensing devices (eg, wearable accessories or patches, etc.) in addition to the operator interface, but additional sensing devices add cost on the one hand and are not. Daily wear habits.

Therefore, how to solve the problems of the current control of automated machine equipment and provide precise and convenient control technology is the purpose of developing the present invention.

In view of various problems of the prior art, the present invention provides a control device including a detection module, a processing module, and a transmission module. The detection module is configured to capture the dynamic image as the target image to display the target image in a webpage; the processing module is configured to receive the target image, convert the target image into a data command; and transmit the module The data module is received from the processing module to transmit the data command to a machine device via the network.

The present invention provides a control method, including: capturing a motion image as a target image to present the target image in a web page; converting the target image into a data command; and transmitting the data command to the network via the network mechanical equipment.

In an embodiment, the initialization module is configured to log in to the webpage to obtain initialization data for controlling the machine device. For example, the initialization data for controlling the machine device is obtained before the motion picture is captured.

In one embodiment, the control device further includes a display unit for displaying the web page. For example, after transmitting the data command to the machine device via the network, the mobile device receives motion information corresponding to the data command from the network to display the motion image of the machine device in a display unit.

In one embodiment, the detection module captures the motion image as a mode of the target image, including: a skin color detection mode, a convex hull detection mode, and/or a convex defect detection mode.

The invention further provides a control system comprising: a control device and a machine device. The control device comprises: a detection module for capturing a dynamic image as a target image to present the target image in a webpage; and a processing module for receiving the target image and converting the target image into data And the transmission module receives the data command from the processing module. The machine device receives data commands of the transmission module transmitted via the network, causing the machine device to execute the data command and generate motion information corresponding to the data command.

In an embodiment, the transmission module is capable of receiving the motion information via the network.

In one embodiment, the machine includes a main controller and a robot for executing the data command to control the robot and The motion information is transmitted to the control device via the network.

The invention realizes the somatosensory detection and the data operation on the user end based on the control technology of the user end. When the control device and method of the present invention are used to control the manipulator, the body feeling detection can be performed without operating buttons and additional sensing devices, and the data packet is not uploaded to the server operation processing, which not only solves the cumbersome positioning of the current robot, The problem of packet loss and transmission time-consuming, the separate window can also display the somatosensory detection, control data and instant image for the operator to directly compare the effects, and greatly improve the cleanliness, convenience and immediacy of the robot control. accuracy.

1‧‧‧Control machine

10‧‧‧ Operator

100‧‧‧Network

11‧‧‧ button

2,61‧‧‧Control device

20,62‧‧‧ Machines

201, 202, 203, 204, 701, 702, 703, 704, 705 ‧ ‧ steps

21,611‧‧‧Initialization module

22,612‧‧‧Test module

23,613‧‧‧Processing module

24,614‧‧‧Transmission module

25,616‧‧‧ display unit

30,40,50‧‧‧motion images

301, 401, 501, 82 ‧ ‧ target image

302,402,502‧‧‧Contrast images

303, 403‧‧‧ surrounding images

6‧‧‧Control system

615‧‧‧Photography module

617‧‧‧ storage unit

620‧‧‧Main control machine

621‧‧‧Master controller

622‧‧‧manipulator

623‧‧‧Monitoring module

81‧‧‧Analog image

811‧‧‧ center point

821‧‧‧ scheduled mobile location

83‧‧‧ sports images

L1‧‧‧ convex hull

P1‧‧‧ Center Point

P2‧‧‧ recessed point

1 is a perspective view of a conventional control machine; FIG. 2 is a perspective view of a control device of the present invention; FIG. 3 is a functional block diagram of a control device of the present invention; and FIG. 4 is a first embodiment of a control method of the present invention; Step 5A is a schematic diagram of capturing a target image by using a skin color detection mode; FIG. 5B is a schematic diagram of capturing a target image by using a skin color detection mode in combination with a convex hull detection mode; and FIG. 5C is a combination of a skin color detection mode, A schematic diagram of a convex envelope detection mode and a convex defect detection mode for capturing a target image; 6A and 6B are block diagrams of different embodiments of the control system of the present invention; and FIG. 7 is a second embodiment of the control method of the present invention. Step flow chart; Figures 8A to 8C are schematic diagrams showing the motion images of the robot corresponding to the target image of the operator.

The embodiments of the present invention are described by way of specific examples, and those skilled in the art can understand the advantages and advantages of the present invention as disclosed in the present disclosure. The present invention may be embodied or applied in various other specific embodiments, and various modifications and changes may be made without departing from the spirit and scope of the invention.

The singular forms "a", "the", and "the"

In the present invention, the term "node-webkit" is an instant running environment that integrates both text of node.js and webkit, webkit provides a document object model (DOM), and node.js provides local language service, user Native applications can be written in HTML5, CSS3, and Javascript. The term "node.js" is based on the platform established by Chrome Javascript. The event-driven I/O server-side Javascript environment allows users to write scalable network programs for instant data-intensive distributed devices. application. The term "WebRTC" (Web Real-Time Communication) is an application programming interface (API) that supports web browsers for instant voice or video conversations. Javascript provides instant web-based instant messaging, including voice. Video capture, decoding, network transmission Loss, display and other functions, and cross-platform support for windows, linux, mac, android. The term "canvas" is part of HTML5. It uses Javascript to draw images on a web page. It can control each element in the canvas area, and has a variety of drawing paths, shapes, characters, and images. The term "UDP" (User Datagram Protocol) is an Open System Interconnection Reference Model (OSI model) that does not require a transport layer protocol.

The invention provides a control device and a method based on remote control of a user, which does not need to transmit data to a server for calculation, reduces the probability of packet loss and data packet transmission time, and can be used across platforms. The term “remote” as used herein is not at the site where the machine is located, for example, the operator and the machine are located in different rooms, different regions or different countries.

As shown in FIGS. 2, 3, and 4, the control device 2 includes a display unit 25, an initialization module 21, a detection module 22, a processing module 23, and a transmission module 24.

The remote control of the control device 2 by the event triggering includes the following steps: Step 201: The operator 10 can use the initialization module 21 to log in to the webpage of the present invention (displayed on the display unit 25) via the network 100 to obtain The initialization data of a machine device 20 is controlled. In step 202, the detection module 22 captures the motion image of the operator 10 as a target image, and parses the continuous primitive data corresponding to the target image. Step 203: The processing module 23 receives the initialization data and the continuous primitive data from the initialization module 21 and the detection module 22, and converts the continuous primitive data into a data instruction according to the initialization data. Step 204, the transmission module 24 is from the processing module Group 23 receives the data command and transmits the data command to the machine device 20 via network 100. The sequence of step 201 and step 202 may be determined according to requirements, or may be performed simultaneously.

In the present embodiment, the machine device 20 is a main control machine 620 for injection molding machines (providing the main controller 621 of FIG. 6A) and a robot 622, and the operator 10 can be controlled by the control device 2 of the present invention. The gesture action of the operator 10 is taken to perform the positioning of the remote control robot 622.

Specifically, the control method of the present invention supports the running structure of the web browser, and can be used across platforms, as long as the display unit 25 and the communication electronic device for browsing the webpage function, such as a personal computer, a tablet computer, a smart phone, etc. Therefore, the control device 2 based on the user's remote control can be realized without an additional sensing device or uploading the data packet to the server operation processing.

Furthermore, when the operator 10 uses the control device 2 to start remote control, due to the design of the initialization module 21, the initialization data can be obtained by logging in to the webpage of the present invention, which is based on the node-webkit architecture. Cross-platform application.

The detection module 22 receives the motion image of the operator 10 via an external or built-in camera module, and captures the motion image of the operator 10 from the background image as a target image, which may include but is not limited to The gesture of the operator 10, the facial expression of the limb, and the like. In the present invention, one or a combination of multiple recognition computing techniques, such as a skin color detection mode, a convex hull detection mode, a convex defect detection mode, etc., may be used to separate the background from the motion image of the operator 10 and capture the operator's Gesture image.

Figure 5A is a schematic diagram of capturing a target image in a skin color detection mode. As shown in FIG. 5A, the display unit 25 for displaying the web page displays the motion image 30 of the operator 10 and its surroundings. The detection module 22 detects that the color in the motion image 30 is similar to the color region of the skin color. After the confirmation, the color region corresponding to the skin color is captured as the target image 301, and the continuous primitive data corresponding to the target image 301 is parsed. The display unit 25 can synchronously display the contrast image 302 corresponding to the target image 301 in the motion image 30 of the operator 10 according to the continuous element data.

In detail, the skin color detection mode includes an image grayscale algorithm, a skin color probability bar graph, threshold filtering, and RGB coloring. The image grayscale algorithm uses the maximum value of the RGB three components of the primitive in the motion image as the gray value of the primitive, and converts the color primitive data into grayscale primitive data. The skin color probability bar graph describes the proportion of different colors in the entire image, regardless of the spatial position of each color. The color bar graph is particularly suitable for describing images that are difficult to auto-segment. Threshold filtering is mainly used for the segmentation of the long picture. The RGB coloring changes the color of the primitive for each RGB value of each primitive.

In the present embodiment, since the moving image 30 includes surrounding images 303 of various conditions (the illustration is blank, but actually the space environment in which the operator 10 is located), the area of the contrast image 302 is designed on the lower right side of the webpage. The target image 301 captured in a single presentation.

Furthermore, the contrast image 302 can also be used to confirm the accuracy of the skin color detection mode capturing the target image 301. Since the object of the skin color may appear in the surrounding image 303, it is necessary to confirm whether the image is correctly captured by comparing the image 302. Target image 301.

FIG. 5B is a schematic diagram of capturing a target image by using a skin color detection mode in combination with a convex hull detection mode. As shown in FIG. 5B, the detection module 22 can detect the target image 401 in the motion image 40 in combination with the skin color detection mode and the convex hull detection mode. The detection module 22 determines, after the image gradation and the skin color probability bar graph, that the color of the motion image 40 is close to the color region of the skin color, and further selects a plurality of feature points from the motion image 40, and the feature points are The primitive data input scribing function draws a convex hull L1 covering the target contour, calculates the coordinate coordinates of the center point of the feature points of the feature points, and inputs a square function to generate a 3X3 center point P1, and captures the convex hull The L1 primitive area is the target image 401, and the convex hull L1 and the center point P1 can move with the target image 401. The display unit 25 synchronously displays the contrast image 402 corresponding to the target image 401.

In detail, the convex hull detection mode includes not only image graying, skin color probability bar graph, but also Codebook image segmentation algorithm, IPAN contour algorithm, Camshift algorithm, drawing rectangle and line drawing algorithm function. . The basic idea of the Codebook image segmentation algorithm is to obtain a time series model of each primitive, which can handle time fluctuations well. The Codebook image segmentation algorithm creates a CodeBook (CB) structure for each primitive of the current image, and each CodeBook structure is composed of multiple CodeWords (CWs). The IPAN contour algorithm uses a triangle abp to describe the point p. The basic idea of the Camshift algorithm is to perform the MeanShift operation on all the frames in the video image, and the result of the previous frame (ie, the center and size of the search window Search Window) as the next frame MeanShift The initial value of the algorithm's Search Window is thus repeated.

In this embodiment, the target image 401 and its surrounding image 403 in the dynamic image 40 are separated by a Codebook image segmentation algorithm; each point in the convex hull polygon is described by an IPAN contour algorithm; and the target is tracked by the Camshift algorithm. The point at which the image 401 moves; the display of the target point is performed by drawing a rectangle and drawing a rectangular algorithm function.

The 5C figure is a schematic diagram of capturing a target image by combining a skin color detection mode, a convex hull detection mode, and a convex defect detection mode. As shown in FIG. 5C, the convex defect detection mode is to calibrate a plurality of concave points P2 of the convex hull L1 in the dynamic image 50, and input a rectangular function to generate a convex concave element region by using the primitive coordinate data of the concave point P2. Then, the convex recessed element area of the convex shape is taken as the target image 501.

In detail, the convex defect detection mode includes image grayscale algorithm, skin color probability bar graph, CodeBook image segmentation algorithm, IPAN contour algorithm, Camshift algorithm, and calibrated by Delaunay triangulation. The concave point P2 of the plurality of convex depressions is input, and the coordinate data of the concave points P2 is input into the rectangular function to generate a 6×6 square concave point P2, and the convex and convex L1 and the convex center in the front and rear frame primitive data are generated. The target image 501 is taken out by the change of the point P1 and the recessed point P2. The display unit 25 can synchronously display the contrast image 502 corresponding to the target image 501.

It is worth mentioning that the operator can select one of the above three detection modes (such as Figure 5A), two (such as Figure 5B) or three (such as Figure 5C) to capture the target images 301, 401, 501. Each detection mode does not affect the effect of each other. Furthermore, the control device 2 of the present invention can utilize separate window synchronization The motion images 30, 40, and 50 of the operator 10 and its surroundings and the contrast images 302, 402, and 502 corresponding to the operation of the operator 10 are displayed, so that the operator 10 can easily confirm the detection effect.

In this embodiment, after the target images 301, 401, and 501 are captured from the motion images 30, 40, and 50, the detection module 22 analyzes the consecutive images of the corresponding target images 301, 401, and 501 according to the steps of FIG. Metadata. Then, the processing module 23 receives the initialization data and the continuous metadata from the initialization module 21 and the detection module 22, and converts the continuous metadata according to the initialization data into data commands for controlling the device 20. The transmission module 24 then receives the data command from the processing module 23 and transmits the data command to the machine device 20 via the network 100 for remote control of the machine device 20.

According to the above description, the actual situation of the present invention is, for example, when the operator 10 moves the finger by 10 cm in front of the display unit 25, and the detection module 22 transmits the moving distance to the processing module 23, via the processing module. In the operation of group 23, the transmission module 24 transmits the data command to the machine device 20 via the network 100, and the robot 622 of the machine device 20 is displaced by 1 cm (cm) or 1 mm (mm) (conversion result) The position of the robot of the machine device 20 can be adjusted to adjust the position of the X, Y, and Z axes of the robot.

The present invention provides a control system 6, as shown in FIG. 6A, which includes a control device 61 and a machine device 62.

The control device 61 includes a detection module 612, a processing module 613, a transmission module 614, and a display unit 616. Detection module 612 capture operation The author's motion image is used as the target image (for example, the operator's gesture) to parse the continuous element data corresponding to the target image. The processing module 613 receives the initialization data and the continuous primitive data, and converts the continuous primitive data into data instructions according to the initialization data. The transmission module 614 transmits the data command to the machine device 62 via the network 100.

The machine device 62 includes a main controller 621 and a robot 622. The main controller 621 logs into the webpage of the present invention via the network 100 to obtain an initialization data package based on the node-webkit architecture, and converts the initialization data packet into a corresponding instruction. The robot 622 autonomous controller 621 receives the corresponding command to complete the initialization and returns the initialization completion signal to the main controller 621. The main controller 621 returns the initialization completion signal to the webpage of the present invention to notify the operator of the control device 61. The machine device 62 is initially operated.

It should be noted that, after the initialization of the machine device 62 is completed, the web page of the present invention actively sends an acknowledgment command to the main controller 621 at intervals to ensure the connection between the control device 61 and the machine device 62.

In this embodiment, the control device 61 further includes an initialization module 611, a camera module 615, and a storage unit 617. The initialization module 611 obtains initialization data for controlling the device 62 from the network 100. The photography module 615 transmits the motion image of the operator to the detection module 612, and the storage unit 617 is configured to store the initialization data.

As shown in FIG. 6B, the machine device 62 further includes: a monitoring module 623 coupled to the main controller 621 for obtaining motion information of the data command transmitted by the robot 622 corresponding to the control device 61. The monitoring module 623 can be selected from a camera, a photoelectric or an electromagnetic sensing component, but is not limited thereto.

When the transmission module 614 of the control device 61 transmits a data command to the machine device 62 via the network 100, the main controller 621 of the machine device 62 executes a data command to control the robot 622. The monitoring module 623 obtains the motion information of the robot 622 corresponding to the data command and transmits the motion information to the main controller 621. The main controller 621 returns the motion information of the robot 622 to the control device 61 via the network 100. After receiving the motion information of the robot 622, the control device 61 converts the motion information into a motion image corresponding to the data command by the robot 622, and the display unit 616 displays the motion image corresponding to the data command by the robot 622.

Figure 7 is a flow chart showing the steps of the control method of the control system 6 of the present invention. The operator logs in to the webpage and starts to remotely control a machine device. The remote control method includes: Step 701: Obtain initialization data for controlling the machine device; Step 702: Capture the operator's motion image as the target image; Step 703: Converting the target image into a data command according to the initialization data; Step 704: transmitting the data command to the machine device via the network and receiving motion information corresponding to the data command of the device (refer to the foregoing); Step 705: displaying the The target image and the moving image of the machine device corresponding to the data command. The sequence of steps 701 and 702 may be determined according to requirements, or may be performed simultaneously.

In this embodiment, step 701 includes: determining whether the initialization device has been stored by the control device 61 and the device 62. If the determination is no, the initialization data is obtained by logging in to the webpage of the present invention, so that the control device 61 and the device are 62 completes the initialization step. The initialization data is a cross-platform application based on the node-webkit architecture.

Step 702 includes: the control device 61 captures the target image in the motion image in the skin color detection mode, the convex hull detection mode, and/or the convex defect detection mode.

Step 703 includes: the control device 61 parses the continuous primitive data corresponding to the target image, and converts the continuous primitive data into UDP. In detail, each UDP contains: the data packet has: head and tail indication, basic protocol and data instruction content. According to the function of the data instruction, UDP includes: real-time data, motor parameter setting data, alarm, curve data and motion commands. The action command type UDP includes a data command for the main controller to manipulate the movement of the robot along the X and Y axes.

Step 704 includes the control device 61 transmitting a UDP to machine device 62 indicating the "request" of the packet header via the network 100, wherein the "request" UDP includes data commands that can be used to control the robot 622. After the main controller 621 of the machine device 62 executes the data command of the control robot 622, the main controller 621 transmits the UDP of the data packet header "feedback" to the control device 61, and the UDP of the "feedback" includes the robot 622 corresponding to the data command. Sports information.

Step 705 includes: the control device 61 converts the motion information into a motion image corresponding to the data command by the robot 622, and the center point of the robot 622 corresponds to the center point of the target image of the operator, and the motion image is compared with the target image. Superimposed.

The display unit 616 can select the target image of the operator, the moving image of the robot, or simultaneously display the superimposed image of the moving image and the target image to provide an operator with a sense of control on the spot. specifically, As shown in FIGS. 8A to 8C, the target image of the operator is presented on the web page.

As shown in FIG. 8A, when the remote control is performed, the screen of the robot 622 is transmitted to the display unit 616 by the camera, so that the display unit 616 presents the analog image 81 of the robot 622, and the analog image 81 can be set with The center point 811, so that the target image 82 of the operator can be aligned by the center point 811. As shown in FIG. 8B, the target image 82 and the analog image 81 are superimposed to form a synchronous moving motion image 83. . Therefore, by designing the target image 82 and the analog image 81 to perform the positioning operation, the operator can more clearly know the synchronization status of the robot 622 and the human hand. The coordinate page or the alignment condition (not shown) may be displayed on the webpage presented by the display unit 616 to provide operator alignment information.

In addition, the machine device 62 can also transmit the position information of the predetermined movement of the robot 622 (for example, the relative coordinate data of the object to be grasped) to the control device 61. As shown in FIG. 8C, the display unit 616 can display a page in the webpage. The relative distance between the predetermined moving position 821 and the target image 82 of the operator is provided to provide a more intuitive and accurate sense of control by the operator. For example, if the robot 622 needs to move 5 cm or move to a certain position, the distance between the predetermined moving position 821 and the center point 811 represents the distance or position required for the robot 622 to move, so the operator only needs to move the hand. When the movement (in this case, the moving image 83 on the webpage moves), when the center point 811 is moved to overlap the predetermined movement position 821, the robot 622 completes the movement of moving 5 cm or moving to the predetermined position.

In summary, the present invention implements a control technique based on remote control of a user. Surgery, real-time detection and data calculation at the user end. When the control device, system and method of the present invention are used to control the robot at the remote end, the body feeling detection can be performed without operating the button or the additional sensing device, and the data packet is not uploaded to the server operation processing, and the current robot is not only solved. The problem of cumbersome positioning, loss of data packets and time-consuming transmission, the separate window can simultaneously display the somatosensory detection, control data and instant images for the operator to directly compare the effects, and greatly improve the cleanliness and convenience of the robot control. Immediacy and accuracy.

The above embodiments are illustrative only and are not intended to limit the invention. Modifications and variations of the above-described embodiments can be made by those skilled in the art without departing from the spirit and scope of the invention. Accordingly, the scope of the invention should be construed as being included in the scope of the appended claims.

2‧‧‧Control device

20‧‧‧ Machines

21‧‧‧Initialization module

22‧‧‧Test module

23‧‧‧Processing module

24‧‧‧Transmission module

100‧‧‧Network

Claims (11)

A control device includes: a detection module for capturing a motion image as a target image to present the target image in a webpage; and a processing module for receiving the target image and converting the target image to The data command; and the transmission module receives the data command from the processing module to transmit the data command to a machine device via the network. For example, the control device described in claim 1 includes an initialization module for logging in to the webpage to obtain initialization data for controlling the machine device. The control device according to claim 1, wherein the display unit for displaying the web page is further included. The control device of claim 1, wherein the detection module captures the motion image as a mode of the target image, including: a skin color detection mode, a convex hull detection mode, and/or a convex defect detection mode. A control system includes: a control device comprising: a detection module for capturing a dynamic image as a target image to present the target image in a web page; and a processing module for receiving the target image, Converting the target image into a data command; and transmitting the module, receiving the data finger from the processing module And a machine device receiving a data command of the transmission module transmitted via the network, causing the machine device to execute the data command to generate motion information corresponding to the data command. The control system of claim 5, wherein the transmission module is capable of receiving the motion information via the network. The control system of claim 5, wherein the machine device comprises a main controller and a robot, the main controller is configured to execute the data command to control the robot, and the motion information is The network is transmitted to the control device. A control method includes: capturing a motion image as a target image to present the target image in a web page; converting the target image into a data command; and transmitting the data command to a machine device via a network. For example, the control method described in claim 8 of the patent application includes obtaining initialization data for controlling the machine device before extracting the motion picture. The control method as described in claim 8 is further included after receiving the data command to the machine device via the network, and receiving, from the network, motion information corresponding to the data command of the machine device to display the machine The motion image of the device is in a display unit. The control method of claim 8, wherein the mode of capturing the motion image as the target image comprises: a skin color detection mode , convex hull detection mode and/or convex defect detection mode.