WO2022220208A1 - データ処理装置、データ処理方法、プログラム及びデータ処理システム - Google Patents

データ処理装置、データ処理方法、プログラム及びデータ処理システム Download PDF

Info

Publication number
WO2022220208A1
WO2022220208A1 PCT/JP2022/017459 JP2022017459W WO2022220208A1 WO 2022220208 A1 WO2022220208 A1 WO 2022220208A1 JP 2022017459 W JP2022017459 W JP 2022017459W WO 2022220208 A1 WO2022220208 A1 WO 2022220208A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
captured image
angle
robot
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/017459
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
ジェイド ファウンテン
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Telexistence Inc
Original Assignee
Telexistence Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telexistence Inc filed Critical Telexistence Inc
Priority to US18/286,132 priority Critical patent/US12417572B2/en
Priority to CN202280027819.XA priority patent/CN117203973A/zh
Priority to JP2023514637A priority patent/JP7803476B2/ja
Priority to KR1020237031401A priority patent/KR20230172460A/ko
Publication of WO2022220208A1 publication Critical patent/WO2022220208A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/06—Control stands, e.g. consoles, switchboards
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
    • B25J19/02—Sensing devices
    • B25J19/021—Optical sensing devices
    • B25J19/023—Optical sensing devices including video camera means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J3/00—Manipulators of leader-follower type, i.e. both controlling unit and controlled unit perform corresponding spatial movements
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1679—Program controls characterised by the tasks executed
    • B25J9/1689—Teleoperation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G06T11/60—Creating or editing images; Combining images with text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present disclosure relates to a data processing device, a data processing method, a program, and a data processing system.
  • Patent Document 1 discloses a technique for displaying a captured image generated by an imaging device provided on a robot on a terminal used by an operator who remotely operates the robot.
  • the present disclosure has been made in view of this point, and aims to provide a data processing device or the like that makes it easier for the operator of the robot to operate the robot.
  • a data processing device acquires captured image data generated based on imaging by an imaging device provided in a robot, and obtains at least part of the captured image data acquired at a first time point.
  • a synthesized image having a wider angle of view than the display image data by synthesizing the display image data and the past captured image data of at least a part of the captured image data acquired at the second time before the first time.
  • a controller configured to generate the data and display the composite image data on a display visible to an operator of the robot.
  • a data processing device or the like is provided that makes it easier for the operator of the robot to operate the robot.
  • FIG. 4 is a diagram showing the relationship between captured image data and synthesized image data; It is a figure which shows the structure of a data processor. It is a figure which shows the method in which a memory
  • 4 is a sequence diagram showing the flow of processing of the data processing system; FIG. 4 is a flow chart showing the flow of processing in the data processing device; It is a figure for demonstrating operation
  • FIG. 1 is a diagram showing an outline of a data processing system S.
  • the data processing system S is a system that provides a telexistence environment in which the operator U can operate an object in real time while making the operator U feel as if an object in a remote location were nearby. be.
  • the data processing system S has a data processing device 1, an operation device 2, and a robot 3.
  • the data processing device 1 is a device that controls the robot 3 according to the operation performed by the operator U on the operation device 2, and is, for example, a computer.
  • the data processing device 1 may be installed in a room where either the operator U or the robot 3 is present, or may be installed at a location different from the location where the operator U and the robot 3 are located.
  • the operating device 2 is a device worn by the operator U, and has a display device 21, an operating device 22, and a communication section 23.
  • the display device 21 has a display on which the operator U can visually recognize an image based on the composite image data generated by the data processing device 1, and is, for example, a goggle-type display device.
  • the image displayed by the display device 21 is, for example, a celestial image corresponding to 360 degrees (all directions).
  • the robot 3 operates based on control data received from the data processing device 1 via the network N.
  • the robot 3 has an imaging device (hereinafter referred to as “camera 31 ”) that generates captured image data, and an image transmission unit 32 that transmits the captured image data generated by the camera 31 to the data processing device 1 .
  • the robot 3 sends at least one robot state data of tactile sense data indicating the tactile sense detected by the robot 3, sound data indicating sounds collected by the robot 3, and joint state data indicating the state of the joints of the robot 3. It is transmitted to the operation device 2 via the data processing device 1 .
  • the robot 3 according to the present embodiment performs a task of arranging products on shelves according to the operation of the operator U in a store where a large number of products are sold on shelves. The place where 3 is provided and the contents of the work are arbitrary.
  • the data processing device 1 acquires captured image data generated by the robot 3 capturing an image of the front using the camera 31 , and displays the acquired captured image data on the display device 21 of the operation device 2 . Since the angle of view of the captured image data generated by the robot 3 is narrower than the field of view of the operator U, if only the captured image data is displayed on the display device 21, the operability of the operator U is poor. Therefore, the data processing device 1 synthesizes captured image data generated by the robot 3 and past captured image data generated in the past corresponding to the surrounding area of the captured image data corresponding to the captured image data. Generate synthetic image data. The operability of the operator U can be improved by causing the data processing device 1 to display the synthetic image data on the operation device 2 .
  • the display device 21 has a sensor that detects the angle of the display device 21 with respect to the reference orientation (hereinafter sometimes referred to as "apparatus angle").
  • the reference orientation is, for example, the orientation of the display device 21 when the display device 21 is worn by the operator U who faces a predetermined orientation. Orientation.
  • the display device 21 detects the angular difference between the reference orientation and the orientation of the display device 21 as the device angle.
  • the device angle of the display device 21 is represented, for example, by a combination of two angles in a spherical coordinate system within a three-dimensional space.
  • FIG. 2 is a diagram for explaining the device angle of the display device 21. As shown in FIG. The device angle at position G on the surface of the sphere shown in FIG. 2 is expressed as ( ⁇ , ⁇ ). ⁇ is the angle between the X-axis direction and a straight line connecting the origin O and the position H obtained by projecting the position G onto the XY plane. ⁇ is the angle between the Z-axis direction and the direction of the straight line connecting the origin O and the position G;
  • the positive direction of the X-axis direction corresponds to the front direction of the display device 21 (that is, the front direction of the operator U) when the display device 21 is activated, and the operator U is facing the front.
  • the device angle of the display device 21 is expressed as (0, 0).
  • the device angle of the display device 21 is expressed as (90, 0) when the operator U faces the true left (that is, the positive direction in the Y-axis direction).
  • the display device 21 generates head operation data indicating the device angle of the display device 21 with respect to the reference orientation.
  • the display device 21 notifies the communication unit 23 of the head operation data at predetermined time intervals.
  • the predetermined time interval is determined, for example, based on the speed at which the robot 3 can change the head angle. Predetermined time intervals are shortened. Since the interval at which the display device 21 notifies the device angle corresponds to the speed at which the robot 3 can change the angle of the head, it is not necessary to detect the device angle more frequently than necessary. Power consumption of the operation device 2 can be suppressed.
  • the operation device 22 is a device used by the operator U to operate the robot 3 with the hands and arms, and has sensors for detecting the movements of the operator U's hands and arms.
  • the operation device 22 generates hand-arm operation data indicating the movements of the operator's U hand and arm.
  • the operation device 22 notifies the communication unit 23 of the generated hand/arm operation data.
  • the operation device 22 may receive an operation for moving the robot 3 forward and backward or left and right.
  • the operation device 22 also has elements for generating heat, pressure, vibration, or the like corresponding to the state of the robot 3 so that the operator U can sense the state of the robot 3 .
  • the communication unit 23 has a communication controller that transmits operation data based on the content of the operation by the operator U to the data processing device 1 and receives composite image data from the data processing device 1 .
  • the communication unit 23 synchronizes the head operation data notified from the display device 21 and the hand-arm operation data notified from the operation device 22, and outputs predetermined operation data including the synchronized head operation data and hand-arm operation data. is transmitted to the data processing device 1 at time intervals of .
  • the communication unit 23 also inputs the composite image data received from the data processing device 1 to the display device 21 .
  • the communication unit 23 may transmit operation data for moving the robot 3 forward, backward, leftward, or rightward to the data processing device 1 .
  • the communication unit 23 may be included in either the display device 21 or the operation device 22 , or may be housed in a housing different from that of the display device 21 and the operation device 22 .
  • the robot 3 orients the head of the robot 3 at the same angle as the angle of the display device 21 based on the head control data generated by the data processing device 1 based on the head operation data transmitted from the operation device 2. change. Further, the robot 3 controls the hands and arms of the robot 3 in the same manner as the hands and arms of the operator U based on the hand-arm control data generated by the data processing device 1 based on the hand-arm operation data transmitted from the operation device 2. move.
  • the camera 31 is provided, for example, on the head, and generates captured image data by photographing the front of the robot 3.
  • the image transmission unit 32 has a communication controller for transmitting captured image data via the network N.
  • FIG. 1 illustrates the case where the camera 31 has the image transmission section 32, the portion where the image transmission section 32 is provided is optional.
  • the direction of the optical axis of the camera 31 changes as the direction of the head of the robot 3 changes based on the head control data, and the imaging area changes.
  • the angle of the direction of the straight line connecting the center of rotation of the head of the robot 3 (corresponding to the origin in FIG. 2) and the center position of the region to be imaged with respect to the front direction is referred to as the "imaging angle”.
  • the robot 3 moves the head of the robot 3 so that the angle of the operator U's head indicated by the head operation data (that is, the device angle) matches the imaging angle.
  • the operation device 2 generates operation data according to the movements of the operator U's head and hands and arms, and transmits the operation data to the data processing device 1 .
  • the data processing device 1 generates control data for operating the robot 3 based on the operation data, and transmits the control data to the robot 3 .
  • the robot 3 operates based on control data received from the data processing device 1 .
  • the robot 3 While the robot 3 is operating, it transmits captured image data generated by the camera to the data processing device 1 at predetermined time intervals (for example, 5 millisecond intervals).
  • the robot 3 may transmit the captured image data to the data processing device 1 in association with information indicating the angle of the head of the robot 3 at the time when the captured image data was generated.
  • the data processing device 1 sends the synthetic image data generated by synthesizing the captured image data received from the robot 3 and the past captured image data corresponding to the area around the imaging region of the captured image data to the operation device 2. and the display device 21 displays the composite image data. Since the synthetic image data includes image data of a wider range than the range that the robot 3 can capture, the operator U can see the actual site where the robot 3 is working. can be secured, and the robot 3 can be easily operated.
  • FIG. 3 is a diagram showing the relationship between captured image data and synthesized image data.
  • An image G1 in FIG. 3A is an image based on the display image data generated by the robot 3 at the first point in time, and shows a state in which bottles placed on shelves are lined up.
  • the image G2 and the image G3 in FIG. 3(a) are captured images that are in contact with both sides of the image capturing area of the image data for display among the plurality of past captured image data generated by the robot 3 at the second time before the first time. It is an image based on captured image data corresponding to the region.
  • FIG. 3B shows a composite image obtained by combining the display image data G1, the past image data G2, and the past image data G3. Compared to the case where the operator U sees the image G1 shown in FIG. 3(a), when the operator U sees the composite image shown in FIG. It becomes easier to work with 3.
  • the configuration and operation of the data processing device 1 will be described in detail below.
  • FIG. 4 is a diagram showing the configuration of the data processing device 1.
  • the data processing device 1 has a communication section 11 , a storage section 12 and a control section 13 .
  • the control unit 13 has an operation data acquisition unit 131 , a control data generation unit 132 , an image data acquisition unit 133 , a synthesis unit 134 and a display control unit 135 .
  • the communication unit 11 has a communication interface for transmitting and receiving various data between the operation device 2 and the robot 3 via the network N.
  • the communication unit 11 inputs, for example, operation data received from the operation device 2 to the operation data acquisition unit 131 . Also, the communication unit 11 inputs the captured image data received from the robot 3 to the image data acquisition unit 133 . Also, the communication unit 11 transmits the synthetic image data input from the display control unit 135 to the operation device 2 .
  • the storage unit 12 has storage media such as ROM (Read Only Memory), RAM (Random Access Memory) and SSD (Solid State Drive).
  • the storage unit 12 stores programs executed by the control unit 13 .
  • the storage unit 12 also temporarily stores captured image data received from the robot 3 .
  • the storage unit 12 associates and stores the device angle, which is the angle of the display device 21 when the captured image data acquired by the image data acquiring unit 133 was captured, and the captured image data. Since the angle of the display device 21 is equivalent to the angle of the head of the robot 3, the storage unit 12 may store the captured image data in association with the angle of the head of the robot 3 when the captured image data was generated. good.
  • FIG. 5 is a diagram showing how the storage unit 12 stores captured image data.
  • the device angles ( ⁇ , ⁇ ) of the display device 21 at the time when the robot 3 generated the captured image data are associated with the file name of the captured image data.
  • the captured image data stored in the storage unit 12 in association with the angle of the display device 21 at the time when the received captured image data was generated is updated. be. Since the angle of the display device 21 and the angle of the head of the robot 3 can be angles corresponding to arbitrary directions in the three-dimensional space, the storage unit 12 stores captured image data corresponding to 360 degrees (all directions). is stored.
  • the control unit 13 has, for example, a CPU (Central Processing Unit) as a processor.
  • the control unit 13 functions as an operation data acquisition unit 131 , a control data generation unit 132 , an image data acquisition unit 133 , a synthesis unit 134 and a display control unit 135 by executing programs stored in the storage unit 12 .
  • a CPU Central Processing Unit
  • the operation data acquisition unit 131 acquires operation data indicating the details of the operation performed by the operator U via the communication unit 11 .
  • the operation data acquisition unit 131 acquires, for example, head operation data indicating the angle of the display device 21 and arm operation data indicating the movement of the hands and arms of the operator U.
  • the operation data acquisition unit 131 inputs the acquired operation data to the control data generation unit 132 , the image data acquisition unit 133 and the synthesis unit 134 .
  • the control data generation unit 132 generates control data for operating the robot 3 based on the operation data input from the operation data acquisition unit 131 .
  • the control data generator 132 may generate the control data by, for example, adding or subtracting a predetermined correction value to or from the value indicated by the operation data.
  • the control data generation unit 132 transmits the generated control data to the robot 3 via the communication unit 11 .
  • the image data acquisition unit 133 acquires, via the communication unit 11, imaged image data generated based on imaging by the camera 31 provided on the robot 3.
  • the image data acquisition unit 133 stores the acquired captured image data in the storage unit 12 .
  • the image data acquiring unit 133 stores the captured image data in the storage unit 12 in association with the angle of the display device 21 indicated by the operation data at the time of acquiring the captured image data.
  • the image data acquisition unit 133 After acquiring the captured image data, the image data acquisition unit 133 acquires the captured image data stored in the storage unit 12 in association with the angle of the display device 21 indicated by the operation data at the time when the captured image data was captured.
  • the captured image data acquired by the acquisition unit 133 is updated. That is, when the image data acquiring unit 133 acquires captured image data at an angle for which captured image data is already stored in the storage unit 12, the image data acquiring unit 133 replaces the captured image data stored in the storage unit 12 with newly acquired captured image data. update to. Note that when the captured image data is associated with the angle of the head of the robot 3 at the time the robot 3 generated the captured image data, the image data acquiring unit 133 associates the angle of the head of the robot 3 with the captured image data.
  • the captured image data may be stored in the storage unit 12 .
  • the image data acquisition unit 133 may initialize the captured image data stored in the storage unit 12 when the position of the robot 3 is moved.
  • the image data acquisition unit 133 detects that an operation to move the position of the robot 3 has been performed. Initialize the image data.
  • the image data acquisition unit 133 initializes the captured image data by deleting the captured image data stored in the storage unit 12, for example.
  • the image data acquisition unit 133 may delete the file name of the captured image data associated with the device angle as shown in FIG. 5 without deleting the captured image data.
  • the image data acquisition unit 133 initializes the captured image data, thereby preventing the display device 21 from displaying an inconsistent synthesized image.
  • the image data acquisition unit 133 may initialize the captured image data stored in the storage unit 12 when the operator U finishes operating the robot 3 or when a predetermined time has passed.
  • the predetermined time is a length of time during which the surrounding conditions can change at the position where the robot 3 is located, and is, for example, one minute.
  • the image data acquisition unit 133 may initialize the captured image data stored in the storage unit 12 when a predetermined time associated with at least one of the date and time or the day of the week has passed.
  • the image data acquisition unit 133 receives an instruction to initialize the captured image data stored in the storage unit 12 input by the operator U through the operation device 22, the image data acquisition unit 133 obtains the captured image stored in the storage unit 12. Data may be initialized. By operating the image data acquiring unit 133 in this manner, the captured image data can be initialized according to the operator U's intention when the operator U starts to feel the deterioration of the quality of the synthesized image data.
  • the image data acquisition unit 133 stores data in the storage unit 12 when the operator U performs an operation to log out in order to end the operation of the robot 3 or when the power of the robot 3 is turned off.
  • the captured image data may be initialized.
  • the synthesizing unit 134 generates synthetic image data as shown in FIG. 3(b). Specifically, the synthesizing unit 134 combines at least part of the image data for display of the captured image data acquired by the image data acquiring unit 133 at the first time point and the image data acquired by the image data acquiring unit 133 at the By synthesizing at least a part of the captured image data acquired at the two points in time with the past captured image data, combined image data having a wider angle of view than the display image data is generated. At least part of the image data for display of the captured image data is captured image data corresponding to at least a part of the imaging region corresponding to the captured image data.
  • the image data for display may be the same as the captured image data acquired by the image data acquisition unit 133 at the first time point.
  • the synthesizing unit 134 combines the image data for display based on the captured image data generated by imaging the camera of the robot 3 at a first imaging angle, and the display image data generated by imaging the camera at a second imaging angle different from the first imaging angle.
  • Composite image data is generated by synthesizing past captured image data based on the captured image data.
  • the first imaging angle is, for example, the angle of the head of the robot 3 when the image G1 shown in FIG. 3A is generated
  • the second imaging angle is the image G2 shown in FIG. It is the angle of the head of the robot 3 when the image G3 is generated.
  • the angle of view in the horizontal direction in one captured image data is 30 degrees
  • the first imaging angle corresponding to the image G1 is (0, 0)
  • the second imaging angle corresponding to the image G2 is (30, 0)
  • the second imaging angle corresponding to the image G3 is (330, 0).
  • the synthesis unit 134 determines the relationship between the angle of the display device 21 indicated by the operation data acquired via the operation data acquisition unit 131 and the multiple device angles corresponding to the multiple captured image data stored in the storage unit 12. Based on this, past captured image data is selected from a plurality of past captured image data stored in the storage unit 12 . Specifically, the synthesizing unit 134 selects past captured image data corresponding to a device angle different from the angle of the display device 21 indicated by the operation data by an angle corresponding to the angle of view of the display image data.
  • the synthesizing unit 134 calculates an angle obtained by adding or subtracting the angle of view of the captured image data (for example, 30 degrees) to the angle of the display device 21 indicated by the head operation data, and stores the calculated angle in association with the device angle corresponding to the calculated angle.
  • the past captured image data stored in the unit 12 is selected. As in the above example, when the device angle of the display device 21 is (0, 0), the synthesizing unit 134 adds (30, 0) to (0, 0), which is associated with (30, 0). and the past captured image data on the right associated with (330, 0) obtained by subtracting (30, 0) from (0, 0).
  • the synthesizing unit 134 may further select past captured image data corresponding to the vertical, diagonally upward, and diagonally downward angles of the image data for display.
  • the synthesizing unit 134 selects the captured image data stored in the storage unit 12 in association with the device angle (0, 60) as the upper past captured image data.
  • the synthesizing unit 134 selects the captured image data stored in the storage unit 12 in association with the device angle (0, 300) as the lower previous captured image data.
  • the synthesizing unit 134 selects the captured image data stored in the storage unit 12 in association with the device angle (30, 60) as the past diagonally upper left captured image data.
  • the synthesizing unit 134 selects the device angles or Based on the imaging angle, a plurality of pieces of captured image data to be used for constructing the combined image data are selected.
  • the synthesizing unit 134 creates celestial image data corresponding to 360 degrees (omnidirectional) by synthesizing a plurality of selected captured image data.
  • the synthesizing unit 134 may use an imaging angle corresponding to the angle of the head of the robot 3 instead of the angle of the display device 21 to select past captured image data used to generate synthesized image data. .
  • the synthesizing unit 134 determines the relationship between the first imaging angle at the first point in time when the image data for display is generated and the plurality of imaging angles corresponding to the plurality of captured image data stored in the storage unit 12. Based on this, the past captured image data is selected from the plurality of captured image data stored in the storage unit 12 . Specifically, similarly to the case where the device angle of the display device 21 is used, the synthesizing unit 134 generates a previously captured image corresponding to an imaging angle that differs from the first imaging angle by an angle corresponding to the angle of view of the image data for display. Select data.
  • the synthesizing unit 134 may generate synthetic image data in such a manner that display image data and past captured image data can be discriminated. For example, the synthesizing unit 134 sets the display image data to color image data and sets the past captured image data to monochrome image data, so that the operator U can distinguish between the display image data and the past captured image data. do.
  • the synthesizing unit 134 may generate synthetic image data in which the brightness of the image data for display is higher than the brightness of the previously captured image data.
  • the synthesizing unit 134 may generate synthesized image data indicating a boundary line between the image data for display and the previously captured image data.
  • the synthesizing unit 134 By the synthesizing unit 134 generating such synthesized image data, the operator U can recognize that the past captured image data is the past captured image data. As a result, the operator U can operate the robot 3 assuming that the state indicated by the past captured image data may differ from the current state, thereby improving safety.
  • the display control unit 135 displays the composite image data on the display device 21 that can be visually recognized by the operator U of the robot 3 .
  • the display control unit 135 causes the display device 21 to display the combined image data in a form in which the image data for display and the previously captured image data can be distinguished.
  • the display control unit 135 causes the display device 21 to display the combined image data by transmitting the combined image data generated by the combining unit 134 to the operation device 2 via the communication unit 11 .
  • FIG. 6 is a sequence diagram showing the processing flow of the data processing system S. As shown in FIG. The sequence diagram shown in FIG. 6 starts when the power of the robot 3 is turned on.
  • the robot 3 generates captured image data by photographing the surroundings with the camera 31 .
  • the robot 3 generates captured image data corresponding to various angles while switching the angle of the head (S1). Send to 1.
  • the data processing device 1 stores the captured image data received from the robot 3 in the storage unit 12 (S2).
  • the data processing device 1 generates data as shown in FIG. 5 by, for example, storing captured image data in the storage unit 12 in association with the imaging angle data.
  • the operation device 2 When the operator U turns on the operation device 2 and starts operating the robot 3 , the operation device 2 generates operation data (S 3 ) and transmits the generated operation data to the data processing device 1 .
  • the control data generator 132 generates control data based on the received operation data (S4) and transmits the generated control data to the robot 3.
  • the robot 3 transmits the captured image data to the data processing device 1 .
  • the synthesizing unit 134 identifies the angle of the display device 21 or the angle of the head of the robot 3 when the robot 3 generated the captured image data (S5).
  • the synthesizing unit 134 selects captured image data generated in the past stored in the storage unit 12 based on the specified angle, and combines the latest captured image data acquired from the robot 3 with the captured image generated in the past.
  • Composite image data is generated by combining the data (S6).
  • the display control unit 135 transmits the synthetic image data generated by the synthesizing unit 134 to the operation device 2 .
  • the controller device 2 displays the received composite image data (S7).
  • the robot 3 switches the angle of the head to generate captured image data corresponding to various angles.
  • the robot 3 does not have to switch the angle of the head and generate the captured image data before the operator U starts the operation. In this case, the robot 3 starts generating captured image data after the operator U starts the operation.
  • the synthesizing unit 134 Until captured image data corresponding to various angles are accumulated in the storage unit 12, the synthesizing unit 134 generates captured image data immediately after being generated by the robot 3 (that is, real-time captured image data not stored in the storage unit 12). image data) is generated. After that, when the captured image data starts to be stored in the storage unit 12 , the synthesizing unit 134 generates a synthesized image obtained by synthesizing the captured image data immediately after generated by the robot 3 and the past captured image data stored in the storage unit 12 . Generate data. By operating the synthesizing unit 134 in this way, the operator U does not have to wait until the data processing device 1 stores the captured image data in the storage unit 12, and the work can be started quickly.
  • FIG. 7 is a flow chart showing the flow of processing in the data processing device 1. As shown in FIG. The flowchart shown in FIG. 7 starts when the operating device 2 is powered on.
  • the image data acquisition unit 133 acquires captured image data from the robot 3 via the communication unit 11 (S11). Further, the operation data acquisition unit 131 acquires operation data from the operation device 2 via the communication unit 11 (S12). The image data acquisition unit 133 acquires the operation data from the operation data acquisition unit 131, and determines whether or not the operation data indicates an operation for changing the position of the robot 3 (S13). When the image data acquisition unit 133 determines that an operation to change the position of the robot 3 has been performed (YES in S13), it initializes the past captured image data stored in the storage unit 12 (S14).
  • the synthesizing unit 134 identifies the device angle of the display device 21 or the angle of the head of the robot 3 at the time when the acquired captured image data was generated, and based on the identified angle, the past data to be synthesized. A plurality of device angles corresponding to the captured image data are calculated (S15).
  • the synthesizing unit 134 determines whether past captured image data corresponding to the plurality of calculated device angles is stored in the storage unit 12 (S16). When combining unit 134 determines that past captured image data corresponding to at least one device angle out of a plurality of device angles is stored in storage unit 12 (YES in S16), Previously captured image data corresponding to the device angle is read out of the plurality of previously captured image data stored in the apparatus. The combining unit 134 generates combined image data by combining the read past captured image data with the captured image data acquired in S1 by the image data acquisition unit 133 (S17).
  • the synthesizing unit 134 sets the region corresponding to the device angle to a predetermined pixel value (for example, a black or white pixel value). Generates combined image data.
  • the synthesizing unit 134 causes the synthetic image data to be transmitted to the display control unit 135 (S18). If past captured image data corresponding to all device angles calculated in S15 is not stored in the storage unit 12 (NO in S16), the synthesizing unit 134 does not generate synthetic image data, and the image data acquiring unit The captured image data acquired by 133 is transmitted to the display control unit 135 (S19).
  • the control unit 13 repeats the processes from S11 to S19 until the operator U performs an operation to end the operation on the operation device 2 (NO in S20). If the operator U has performed an operation to end the operation (YES in S20), the control unit 13 ends the process.
  • the synthesizing unit 134 synthesizes the captured image data corresponding to the area captured by the robot 3 with the past captured image data stored in the storage unit 12.
  • the synthesizing unit 134 Display image data generated by cutting out a partial area of captured image data acquired by the image data acquisition unit 133 and a plurality of past captured image data corresponding to areas in contact with the display image data are synthesized. Composite image data may be generated in this way.
  • FIG. 8 is a diagram for explaining the operation of the data processing device 1 in the first modified example.
  • the synthesizing unit 134 generates display image data Gc by cutting out a predetermined partial area from the imaged image data G0 acquired from the robot 3 .
  • the predetermined partial area is an area including the center position of the imaging area of the captured image data G0, which is highly important when the operator U operates the robot 3, for example.
  • the synthesis unit 134 calculates a plurality of device angles corresponding to a plurality of past captured image data to be synthesized with the display image data Gc, based on the angle of view of the display image data.
  • the synthesizing unit 134 reads from the storage unit 12 a plurality of pieces of past captured image data corresponding to the plurality of calculated device angles. Specifically, the synthesizing unit 134 stores the past captured image data that contacts the display image data in the upper, lower, left, right, and oblique directions of the display image data Gc generated by cutting out the captured image data. 12.
  • the combining unit 134 generates combined image data by combining the display image data Gc and a plurality of previously captured image data read from the storage unit 12 (image data of the area indicated by the dashed line in FIG. 8).
  • the synthesizing unit 134 cuts out a partial area from the imaged image data G0 acquired from the robot 3 to generate the display image data Gc in this way, thereby reducing the area of the moving image data that is switched in real time. , the amount of synthesized image data can be reduced. As a result, the delay time in transmitting the composite image data from the data processing device 1 to the operation device 2 is reduced, and the load of display processing on the operation device 2 is reduced.
  • FIG. 9 is a diagram for explaining the operation of the data processing device 1 in the second modified example.
  • the robot 3 transmits the captured image data G0 generated by the camera to the data processing device 1 as it is.
  • the clipped captured image data Gc may be transmitted to the data processing device 1 .
  • the image data acquisition unit 133 acquires captured image data Gc corresponding to a partial area of the area captured by the camera.
  • the captured image data is displayed on the display device 21. delay time can be shortened.
  • the operation device 2 acquires the captured image data transmitted by the robot 3, and based on the device angle corresponding to the acquired captured image data, at least any one of above, below, left, and right of the captured image data.
  • the past captured image data in contact with one side is selected, and combined image data is generated by synthesizing the captured image data and the past captured image data.
  • the display device 21 transmits the head operation data indicating the angle of the head of the operator U, but the display device 21 detects the direction of the operator U's line of sight and You may transmit the head operation data which show an angle with respect to a direction.
  • the reference orientation is the front orientation of the operator U. As shown in FIG.
  • the direction of the line of sight of the operator U is determined based on the direction of the operator U's head with respect to the front of the operator U and the direction of the line of sight with respect to the direction orthogonal to the display screen of the display device 21 .
  • the display device 21 detects an angle displaced from the orientation of the display device 21 when the display device 21 is activated, and an angle between a direction orthogonal to the display screen of the display device 21 and the direction of the line of sight. By adding, the angle between the reference direction and the line-of-sight direction is calculated. For example, when the head of the display device 21 is directed 90 degrees to the right and the line of sight is 30 degrees to the left with respect to the front of the display device 21, the angle between the reference direction and the direction of the line of sight is 60 degrees.
  • the control data generation unit 132 Upon receiving the head operation data indicating the angle between the reference direction and the line-of-sight direction, the control data generation unit 132 generates control data with the angle as the angle of the head, and transmits the generated control data to the robot 3. do. By changing the angle of the head of the robot 3 based on the received control data, the camera 31 can generate captured image data centered on the direction of the operator U's line of sight.
  • the data processing system S By configuring the data processing system S in this way, when the operator U moves his or her line of sight, the captured image data that matches the direction of the viewpoint can be viewed in real time, and the surrounding area of the captured image data can be viewed. Since the user can see the image based on the past captured image data, the operability is further improved.
  • the synthesizing unit 134 combines at least a part of display image data of the captured image data acquired by the image data acquiring unit 133 at the first point of time, and the image data acquiring unit Composite image data having a wider angle of view than display image data is generated by synthesizing at least a part of the captured image data acquired at a second time point earlier than the first time point with past captured image data. Then, the display device 21 displays an image based on the synthesized image data generated by the synthesizing section 134 .
  • the operator U can work while viewing the captured image in a wider range than the range that the robot 3 can capture, thereby improving workability and Improves safety. Since the data processing system S can reduce the range to be imaged by the robot 3, the area of the image pickup device of the robot 3 can be reduced to reduce the cost, or the image pickup device with the same image pickup area can be used to increase the resolution. It is also possible to improve it.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Manipulator (AREA)
  • Numerical Control (AREA)
PCT/JP2022/017459 2021-04-14 2022-04-11 データ処理装置、データ処理方法、プログラム及びデータ処理システム Ceased WO2022220208A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US18/286,132 US12417572B2 (en) 2021-04-14 2022-04-11 Data processing apparatus, data processing method, program, and data processing system
CN202280027819.XA CN117203973A (zh) 2021-04-14 2022-04-11 数据处理装置、数据处理方法、程序以及数据处理系统
JP2023514637A JP7803476B2 (ja) 2021-04-14 2022-04-11 データ処理装置、データ処理方法、プログラム及びデータ処理システム
KR1020237031401A KR20230172460A (ko) 2021-04-14 2022-04-11 데이터 처리 장치, 데이터 처리 방법, 프로그램 및 데이터 처리 시스템

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-068174 2021-04-14
JP2021068174 2021-04-14

Publications (1)

Publication Number Publication Date
WO2022220208A1 true WO2022220208A1 (ja) 2022-10-20

Family

ID=83640076

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/017459 Ceased WO2022220208A1 (ja) 2021-04-14 2022-04-11 データ処理装置、データ処理方法、プログラム及びデータ処理システム

Country Status (5)

Country Link
US (1) US12417572B2 (https=)
JP (1) JP7803476B2 (https=)
KR (1) KR20230172460A (https=)
CN (1) CN117203973A (https=)
WO (1) WO2022220208A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7761029B2 (ja) * 2023-09-12 2025-10-28 カシオ計算機株式会社 撮影システム、撮影装置、撮影方法及びプログラム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005323310A (ja) * 2004-05-11 2005-11-17 Nippon Telegr & Teleph Corp <Ntt> 視野共有装置、視野移動量入力装置、画像表示装置、撮影範囲投影方法、視野移動量入力装置の制御方法、画像表示装置の制御方法、視野共有装置のプログラム、視野移動量入力装置のプログラム及び画像表示装置のプログラム
JP2012175136A (ja) * 2011-02-17 2012-09-10 Nec Corp カメラシステムおよびその制御方法
JP2013520757A (ja) * 2010-02-24 2013-06-06 アイピープレックス ホールディングス コーポレーション 視覚障害者支援用拡張現実パノラマ
JP2016167688A (ja) * 2015-03-09 2016-09-15 富士通株式会社 画像生成方法、システム、装置、及び端末

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI554976B (zh) * 2014-11-17 2016-10-21 財團法人工業技術研究院 監控系統及其影像處理方法
US11937769B2 (en) * 2017-12-28 2024-03-26 Cilag Gmbh International Method of hub communication, processing, storage and display
JP6666400B1 (ja) 2018-09-10 2020-03-13 Ｔｅｌｅｘｉｓｔｅｎｃｅ株式会社 ロボット制御装置、ロボット制御方法及びロボット制御システム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005323310A (ja) * 2004-05-11 2005-11-17 Nippon Telegr & Teleph Corp <Ntt> 視野共有装置、視野移動量入力装置、画像表示装置、撮影範囲投影方法、視野移動量入力装置の制御方法、画像表示装置の制御方法、視野共有装置のプログラム、視野移動量入力装置のプログラム及び画像表示装置のプログラム
JP2013520757A (ja) * 2010-02-24 2013-06-06 アイピープレックス ホールディングス コーポレーション 視覚障害者支援用拡張現実パノラマ
JP2012175136A (ja) * 2011-02-17 2012-09-10 Nec Corp カメラシステムおよびその制御方法
JP2016167688A (ja) * 2015-03-09 2016-09-15 富士通株式会社 画像生成方法、システム、装置、及び端末

Also Published As

Publication number Publication date
JP7803476B2 (ja) 2026-01-21
JPWO2022220208A1 (https=) 2022-10-20
KR20230172460A (ko) 2023-12-22
US12417572B2 (en) 2025-09-16
CN117203973A (zh) 2023-12-08
US20240185492A1 (en) 2024-06-06

Similar Documents

Publication Publication Date Title
US10095458B2 (en) Information processing apparatus, information processing method, non-transitory computer-readable storage medium, and system
JP4553362B2 (ja) システム、画像処理装置、情報処理方法
JP2001008232A (ja) 全方位映像出力方法と装置
JP2012060216A (ja) パノラマ画像の生成方法
JP7792497B2 (ja) 遠隔操作のための表示システム、表示方法およびプログラム
US11618166B2 (en) Robot operating device, robot, and robot operating method
KR20170062439A (ko) 제어 장치, 제어 방법 및 프로그램
JP3242079U (ja) 浮遊画像表示装置、及び浮遊画像表示システム
WO2021230364A1 (ja) 遠隔制御システムとその遠隔作業装置、映像処理装置およびプログラム
WO2022220208A1 (ja) データ処理装置、データ処理方法、プログラム及びデータ処理システム
JP2005339377A (ja) 画像処理方法、画像処理装置
JP6518645B2 (ja) 情報処理装置および画像生成方法
JP2023075353A (ja) 仮想オブジェクト操作方法およびヘッドマウントディスプレイ
JP7203157B2 (ja) 映像処理装置およびプログラム
WO2024042843A1 (ja) アバター制御装置、アバター制御方法、及びアバター制御プログラム
JP7795163B2 (ja) データ処理装置、データ処理方法及びデータ処理システム
JP2005252831A (ja) 設備監視支援装置
US20250013418A1 (en) Work assistance system
US20250131633A1 (en) Avatar control device, avatar control method, and avatar control program
JP2021022075A (ja) 映像表示制御装置、方法およびプログラム
WO2025243438A1 (ja) 遠隔作業支援装置、遠隔作業支援システム、遠隔作業支援方法、及び遠隔作業支援プログラム
JP2533590B2 (ja) マニピュレ―タ操作ガイダンス立体表示装置
WO2025187547A1 (ja) 遠隔操作システムおよび遠隔操作方法
JP2025162718A (ja) 情報処理装置、情報処理装置の制御方法、及びプログラム
JP2023154916A (ja) 情報処理装置、方法及びプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22788133

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023514637

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18286132

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 202280027819.X

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22788133

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 18286132

Country of ref document: US