WO2020153298A1 - 画像解析システム - Google Patents

画像解析システム Download PDF

Info

Publication number
WO2020153298A1
WO2020153298A1 PCT/JP2020/001715 JP2020001715W WO2020153298A1 WO 2020153298 A1 WO2020153298 A1 WO 2020153298A1 JP 2020001715 W JP2020001715 W JP 2020001715W WO 2020153298 A1 WO2020153298 A1 WO 2020153298A1
Authority
WO
WIPO (PCT)
Prior art keywords
sensor
processor
image
application
image analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/001715
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
水野　公嘉
俊行 黒崎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Interactive Entertainment Inc
Original Assignee
Sony Interactive Entertainment Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Interactive Entertainment Inc filed Critical Sony Interactive Entertainment Inc
Priority to US17/423,935 priority Critical patent/US12103162B2/en
Priority to CN202080009904.4A priority patent/CN113316505B/zh
Publication of WO2020153298A1 publication Critical patent/WO2020153298A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J11/00—Manipulators not otherwise provided for
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • the present invention relates to a system for analyzing image data.
  • Patent Literature 1 discloses a learning method for gait control by a humanoid legged mobile robot.
  • Patent Document 1 discloses a method for reinforcement learning of a stable walking trajectory that enables walking when the robot cannot stably walk on the initially given trajectory.
  • Robot technology is evolving every day.
  • Conventionally, a four-legged walking type pet robot has been the mainstream of commercially available models, but in recent years, humanoid robots that perform complicated actions such as dancing have been distributed. It is expected that the robot will learn by itself to improve existing functions and acquire new functions by improving the arithmetic processing capacity and improving the learning model.
  • the action purpose and action mode of the autonomous robot are determined by the application to be executed.
  • An application that causes a robot to walk autonomously collects and analyzes environmental information around the robot, and calculates the rotational speeds of a plurality of motors that form a robot joint.
  • Each motor is provided with a control microcomputer, and a plurality of microcomputers synchronize with each other to supply electric power according to the calculated rotation speed to the motor.
  • an application whose purpose is to chase a user is to specify the positions of the user and obstacles in space from the camera image, determine the route to follow the user, and set the motor of each robot joint so that the robot moves along the route. Calculate the rotation speed.
  • the rotation speeds of multiple motors are calculated based on accurate image-sensed data, and the motors rotate at the calculated rotation speeds without any time delay, thereby realizing a behavior mode according to the purpose of the behavior. If an attempt is made to give the robot an advanced autonomous action function, the amount of calculation by the application will be large, but even in that case, in order to allow the robot to take action, the image sensing function and/or the motion control function should be immediately changed. It is necessary to build a mechanism to realize this.
  • the image sensing function is not limited to robots at this time, but it is installed in various moving objects such as vehicles driven by people and toys operated by people (for example, radio-controlled automobiles). Therefore, the image analysis technology including the image sensing function is expected not only in the robot field but also in various fields.
  • an image analysis system includes an application processor that processes an application and a sensor processor that acquires and analyzes image data from an image sensor.
  • the application processor acquires the image analysis result by the sensor processor and the posture information for specifying the orientation of the image sensor.
  • any combination of the above-described components and the expression of the present invention converted between a method, a device, a system, a computer program, a recording medium having a computer program readable recorded thereon, a data structure, etc. are also included in the present invention. Is effective as a mode of.
  • FIG. 1 shows a schematic configuration of an entertainment system 1 of the embodiment.
  • the entertainment system 1 includes a robot device 20 and a server device 10.
  • the robot device 20 is a movable body capable of autonomous walking, and is communicably connected to the server device 10 via a network 2 such as the Internet via an access point (AP) 3.
  • AP access point
  • the robot device 20 is configured as a humanoid robot and is owned by a user who is the owner. It is preferable that the robot device 20 can recognize the owner by face authentication by image analysis, voiceprint authentication by voice analysis, or the like. When the robot device 20 recognizes the owner, for example, only the instruction from the owner can be accepted and the owner can act according to the instruction.
  • the robot device 20 has a part similar to that of a human being, and has an outer shape that makes the human being familiar.
  • the robot device 20 has a head, a neck, a trunk (chest, abdomen, and back), upper limbs, and lower limbs, and the upper limbs may have upper arms, forearms and hands, and the lower limbs may have thighs, lower legs and feet.
  • the parts are connected to each other by an actuator.
  • the actuator includes at least a motor arranged in a joint part which is a movable part, and a link mechanism connecting the motors.
  • the robot device 20 drives the actuator to maintain the balance of the posture and exhibits a behavior according to the purpose of the behavior.
  • the robot apparatus 20 realizes basic functions including functions of walking and running and functions of avoiding obstacles by a basic application program (hereinafter also referred to as “basic application”) that describes a control method for each actuator. .. Since the basic application is in charge of the basic functions of the robot device 20, it is preferable that the basic application is preinstalled in the robot device 20, and may be incorporated in middleware, for example.
  • ⁇ Application programs other than basic applications are applied application programs (hereinafter also referred to as "applied applications").
  • the application application implements additional functions such as a dance function.
  • the application application is supplied from the server device 10 as needed and installed in the robot device 20.
  • the robot apparatus 20 acquires a new function by downloading and installing a new application application.
  • FIG. 2 shows the hardware configuration of the control system 4.
  • the control system 4 includes an architecture for realizing the immediacy of the image sensing function and the motion control function of the robot device 20.
  • the control system 4 has a multiprocessor configuration and includes an application processor 30 that processes an application, a sensor processor 50 that processes image data, and a motion control processor 70 that controls the motion of a movable part of the robot apparatus 20.
  • the application processor 30 and the sensor processor 50 build an image analysis system
  • the application processor 30 and the motion control processor 70 build a motion control system.
  • the application processor 30, the sensor processor 50, and the motion control processor 70 are provided in the housing of the robot device 20, and the two processors are connected to each other so that they can communicate with each other.
  • the application processor 30 operates as the main processor.
  • the sensor processor 50 performs the image analysis instructed by the application processor 30 and provides the image analysis result to the application processor 30.
  • the motion control processor 70 drives and controls the actuator at the rotation speed instructed by the application processor 30. Since the sensor processor 50 and the motion control processor 70 perform their respective processes, the application processor 30 does not need to have the image sensing function and the motion control function, and the processing load of the application processor 30 can be reduced.
  • the application processor 30 executes a first operating system (hereinafter, also referred to as “first OS”) to process an application.
  • the application to be processed is both a basic application and an application application, and the application processor 30 simultaneously processes a plurality of basic applications while also processing a necessary application application. Therefore, the application processor 30 has a plurality of CPU cores.
  • the first OS is preferably a general-purpose OS capable of simultaneously executing various applications.
  • a display device 32, a speaker 34, a communication device 36, and a microphone 38 are connected to the application processor 30.
  • the display device 32 and the speaker 34 respectively output the image and the sound generated by the application processor 30 and present the information to the user.
  • the communication device 36 connects to the AP 3 via a wireless LAN and communicates with the server device 10 via the AP 3.
  • the communication device 36 may have a mobile phone communication function.
  • the communication device 36 can download the application and patch file from the server device 10.
  • the microphone 38 collects ambient voice, converts it into a voice signal, and provides the voice signal to the application processor 30.
  • the application processor 30 has a voice recognition function and may determine an action based on a voice instruction from the user.
  • the sensor processor 50 is a processor that performs an image sensing function, and is connected to the image sensor 52, the distance sensor 54, and the event-driven sensor 56.
  • the image sensor 52, the distance sensor 54, and the event-driven sensor 56 are mounted on the robot apparatus 20 and operate as a visual sensor that acquires peripheral information of the robot apparatus 20, and the sensor processor 50 is responsible for a vision recognition function.
  • the image sensor 52 is a camera, and supplies an RGB image taken at a predetermined cycle (for example, 1/60 seconds) to the sensor processor 50.
  • the range sensor 54 which may be a TOF range image sensor, provides a three-dimensional range image to the sensor processor 50.
  • the event-driven sensor 56 is a sensor that detects a change in the brightness value of a pixel of the image sensor, and supplies a combination of the detected time and pixel coordinates to the sensor processor 50.
  • the distance sensor 54 and the event-driven sensor 56 are information sensors that detect information about the subject included in the image data.
  • the event-driven sensor 56 has a wider dynamic range than the image sensor 52, and can accurately detect the movement of the subject even in an environment where incident light exceeding the dynamic range of the image sensor 52 exists. Further, since the event-driven sensor 56 can detect the movement of the subject with a higher time resolution than the image sensor 52, the sensor processor 50, when processing the image data acquired by the image sensor 52, the event-driven sensor 56. It is possible to complementarily use the motion information acquired in.
  • the sensor processor 50 executes a second operating system (hereinafter, also referred to as “second OS”) to process the image data acquired by the image sensor 52.
  • the sensor processor 50 may include a neural network that recognizes an object included in image data using a machine-learned model.
  • the sensor processor 50 acquires depth information of the subject from the distance sensor 54 and movement information of the subject from the event-driven sensor 56, and analyzes image data obtained from the image sensor 52 using these subject information.
  • the application processor 30 instructs the sensor processor 50 on the image analysis function to be executed according to the application to be processed.
  • the sensor processor 50 when the application processor 30 processes an application in which the robot device 20 follows the owner to be followed, the sensor processor 50 is requested to specify the position of the owner. In response to the request, the sensor processor 50 sets the image analysis parameter set for recognizing the owner from the captured image in the neural network, and executes the owner recognition process by the image analysis.
  • the parameter set for owner recognition may be stored in a storage device (not shown) connected to the sensor processor 50, but may be provided from the application processor 30.
  • the sensor processor 50 may execute the analysis process of the image data in consideration of the subject information acquired by the distance sensor 54 and the event-driven sensor 56.
  • the sensor processor 50 may have a learning function for optimizing the parameter set of the neural network in order to realize the image analysis function specialized for the application.
  • the sensor processor 50 preferably performs learning for optimizing the owner recognition parameter set in advance, and optimizes the parameter set before the application processor 30 executes an application that follows the owner. ..
  • the objective function (error function) in the learning function is obtained by comparing the output obtained by inputting the learning image into the neural network and the correct value (that is, the target person is the owner) corresponding to the image. Calculate the error.
  • the learning function calculates the gradient of the parameter by the gradient backpropagation method based on the error, and updates the optimization target parameter of the neural network based on the momentum method. By performing application-specific learning, an optimal parameter set can be generated for each application and an accurate image analysis function will be realized.
  • the sensor processor 50 may have other types of learning functions.
  • the sensor processor 50 performs image analysis for recognizing the specific object while Predict the movement trajectory of. Since the event driven sensor 56 detects the motion information of the subject with high time resolution, the motion information can be suitably used for accurate movement prediction.
  • the sensor processor 50 predicts the movement trajectory of a specific object based on the information from the image sensor 52, the distance sensor 54, and the event-driven sensor 56.
  • the sensor processor 50 grasps the motion of the owner while performing image analysis for recognizing the owner, Furthermore, future movements may be predicted.
  • the sensor processor 50 acquires the movement information of the owner detected by the event-driven sensor 56, grasps the current movement of the owner, and predicts the movement to be performed based on the movement information up to the present. May be.
  • the robot apparatus 20 can move in advance of the movement of the owner.
  • the second OS executed by the sensor processor 50 as described above is preferably an OS for controlling the image analysis neural network.
  • the second OS is a special OS suitable for the neural network, and thus the second OS is not a general-purpose OS but different from the first OS.
  • the motion control processor 70 is a processor that controls the motion of the movable part of the robot apparatus 20, and includes a plurality of motion sensors 72, a plurality of touch sensors 74, and a plurality of microcomputers 76a to 76n (hereinafter referred to as " (Referred to as a microcomputer 76)).
  • One microcomputer 76 is responsible for driving one motor.
  • the motion control processor 70 is in charge of the motion control function of the robot device 20, controls the microcomputer 76, and supplies a drive current to the motor.
  • the motion sensor 72 has a 3-axis acceleration sensor and a 3-axis gyro sensor.
  • the motion sensor 72 may be provided in each joint part and may provide the motion control processor 70 with sensor data indicating a position and a posture of the joint part in a three-dimensional space and/or sensor data indicating a change in the position and the posture.
  • the motion sensor 72 is provided at the position of the image sensor 52, and the motion control processor 70 receives the sensor data indicating the position and the posture of the image sensor 52 in the three-dimensional space and/or the sensor data indicating the change of the position and the posture. May be provided to
  • the touch sensor 74 is provided on the outer surface of the robot apparatus 20 and detects contact with the robot apparatus 20.
  • the motion control processor 70 executes a third operating system (hereinafter, also referred to as “third OS”) to control the motion of the movable part of the robot device 20, that is, the rotation of the motor.
  • the role of the motion control processor 70 is to manage the microcomputer 76 in real time so that the rotation speed of each motor is calculated by the application processor 30, and a more important role is to prevent the robot apparatus 20 from tipping over. .. Therefore, the motion control processor 70 constantly monitors the sensor data of the motion sensor 72 as a premise for responding to the motion request from the application processor 30, and implements the posture control application for the fall prevention.
  • the motion control processor 70 immediately executes motor control for preventing fall when the robot device 20 hits an obstacle and is likely to fall.
  • the third OS executed in the motion control processor 70 is preferably a real-time OS.
  • the real-time OS is an OS that is designed to execute a specific application such as an attitude control application at a highly accurate timing and can process the application in real time.
  • the third OS is different from the first OS and the second OS.
  • the application processor 30, sensor processor 50, and motion control processor 70 are configured as described above.
  • the application processor 30 knows the position in space of the owner and the position in space of an obstacle existing between the robot device 20 and the owner. There is a need. Therefore, the application processor 30 instructs the sensor processor 50 to specify the positions of the owner and the obstacle included in the image data of the image sensor 52.
  • the owner and the obstacle to be recognized by the application processor 30 are also simply referred to as “object”.
  • the sensor processor 50 acquires image data from the image sensor 52, and uses the depth information of the distance sensor 54 and the motion information of the event-driven sensor 56 to perform image analysis instructed by the application processor 30.
  • the sensor processor 50 provides the application processor 30 with the image analysis result including the position coordinates of the object in the camera coordinate system with the image sensor 52 as the base point.
  • the image analysis result also includes depth information (distance information) of the object in the camera coordinate system. Therefore, the application processor 30 can specify the three-dimensional coordinates of the object in the camera coordinate system with the image sensor 52 as the base point.
  • the motion control processor 70 provides the sensor processor 50 with posture information for specifying the orientation of the image sensor 52.
  • This attitude information may include information for specifying the position of the image sensor 52. That is, the posture information includes information for specifying the optical axis direction and position in the space when the image sensor 52 acquires image data.
  • the positions of the joints and the main parts are defined in the robot coordinate system with the reference position in the robot body as the origin.
  • the motion control processor 70 constantly calculates the three-dimensional coordinates of the reference position in the three-dimensional real space coordinate system (spatial coordinate system), and by specifying the orientation and position of the image sensor 52 in the robot coordinate system, the image sensor Posture information for specifying the orientation and position of the 52 in the spatial coordinate system is generated.
  • the sensor processor 50 transmits the image analysis result to the application processor 30 together with the posture information of the image sensor 52.
  • the application processor 30 specifies the orientation and position of the image sensor 52 in the spatial coordinate system by acquiring the posture information, and specifies the three-dimensional coordinates of the image sensor 52 in the camera coordinate system by acquiring the image analysis result. As a result, the application processor 30 acquires the three-dimensional coordinates of the target object in the spatial coordinate system.
  • the motion control processor 70 provides the sensor processor 50 with posture information when the image sensor 52 acquires image data analyzed by the sensor processor 50.
  • the motion control processor 70 may derive the posture information of the image sensor 52 from the sensor data of each motion sensor 72 provided in each joint part.
  • the motion control processor 70 may derive the posture information of the image sensor 52 using the sensor data of the motion sensor 72.
  • the sensor processor 50 notifies the motion control processor 70 of the shooting time of the image data used for analysis, and the motion control processor 70 provides the sensor processor 50 with the posture information of the image sensor 52 at the same time as the shooting time.
  • the motion control processor 70 always executes the posture control application, and calculates the position coordinates of each joint part and the main part at a sampling cycle much shorter than the image analysis cycle (for example, 1/60 second) in the sensor processor 50. doing.
  • the motion control processor 70 of the embodiment derives the posture information of the image sensor 52 in this sampling cycle and stores it in a storage device (not shown) together with time information, and when the image data shooting time is notified, The attitude information of the image sensor 52 at the same time may be read from the storage device and provided to the sensor processor 50.
  • the sensor processor 50 is provided with posture information from the motion control processor 70 during analysis of image data. After finishing the analysis of the image data, the sensor processor 50 adds the posture information of the image sensor 52 as metadata to the image analysis result and provides the image analysis result to the application processor 30.
  • the application processor 30 specifies the position of the target object in the space from the image analysis result and the posture information.
  • the motion control processor 70 of the embodiment provides the posture information of the image sensor 52 to the sensor processor 50 without passing through the application processor 30. That is, the motion control processor 70 directly provides the sensor processor 50 with the posture information via the bus connecting the sensor processor 50.
  • the application processor 30, which is the main processor, relays data transmission between other processors, so the data transmission time is the sum of the relay time by the application processor 30.
  • the control system 4 since the sensor processor 50 and the motion control processor 70 are connected to the bus, when the motion control processor 70 provides the sensor processor 50 with the posture information of the image sensor 52, the motion control processor 70 passes through the application processor 30. Rather than providing it directly to the sensor processor 50, the data transmission time becomes shorter. Therefore, in order to improve the real-time property, the motion control processor 70 directly transmits the posture information of the image sensor 52 to the sensor processor 50.
  • the sensor processor 50 may request the motion control processor 70 to provide posture information of the image sensor 52 during image analysis. At this time, the sensor processor 50 may notify the motion control processor 70 of the shooting time of the image data used in the analysis. Upon receiving the posture information supply request, the motion control processor 70 directly provides the posture information of the image sensor 52 to the sensor processor 50. When the motion control processor 70 is notified of the image data capturing time, thereafter, the motion control processor 70 may periodically provide the posture information according to the image analysis cycle of the sensor processor 50. For example, when the image analysis cycle of the sensor processor 50 is 1/60 second, the motion control processor 70 may provide the posture information to the sensor processor 50 at the same cycle starting from the notified imaging time. The motion control processor 70 may periodically provide the posture information to the sensor processor 50 regardless of the request from the sensor processor 50.
  • the sensor processor 50 adds the posture information as metadata to the image analysis result and provides the image analysis result to the application processor 30. Thereby, the application processor 30 can specify the position of the object in space.
  • the sensor processor 50 provides the image analysis result, but does not provide the image data used in the image analysis to the application processor 30.
  • the transmission delay risk between the sensor processor 50 and the application processor 30 is reduced, and the application processor 30 is provided with the image analysis result without a time delay. To do. Further, since the sensor processor 50 does not provide the image data to the application processor 30, it is possible to prevent the image data including the personal information from leaking to the outside of the robot apparatus 20.
  • the sensor processor 50 may provide the image analysis result to the motion control processor 70.
  • the motion control processor 70 has a predetermined object tracking function
  • the autonomous tracking behavior may be controlled using the image analysis result of the sensor processor 50.
  • the sensor processor 50 and the motion control processor 70 are provided to avoid the situation where the load is concentrated on the application processor 30. Further, each of the application processor 30, the sensor processor 50, and the motion control processor 70 executes an OS suitable for the processing in charge thereof, thereby improving the immediacy of the control system 4 as a whole.
  • the robot device 20 may be any robot that can move, and is not limited to a bipedal robot.
  • the robot device 20 may be an industrial robot or the like.
  • the sensor processor 50 analyzes the captured image data of the image sensor 52 mounted on the robot device 20.
  • the application processor 30 and the sensor processor 50 construct an image analysis system.
  • This image analysis system analyzes a captured image data of an image sensor mounted on a moving body other than the robot apparatus 20. Is also available.
  • the image analysis system may be used to analyze image data of image sensors mounted on various types of moving bodies such as vehicles such as vehicles driven by humans, toys operated by humans (for example, radio-controlled automobiles).
  • the motion control processor 70 provides the posture information of the image sensor 52 to the sensor processor 50, but it may be provided to the application processor 30.
  • the application processor 30 holds the posture information without transferring it to the sensor processor 50, and associates it with the image analysis result provided from the sensor processor 50.
  • the application processor 30 refers to the shooting time of the image data used for the image analysis included in the image analysis result and the acquisition time of the sensor data used for deriving the posture information included in the posture information, and performs the image analysis at the same time.
  • the result and posture information are linked and used to identify the spatial position of the object.
  • the application processor 30, the sensor processor 50, and the motion control processor 70 are provided in the housing of the robot device 20 in the embodiment.
  • the sensor processor 50 and the motion control processor 70 may be provided inside the housing of the robot apparatus 20, and the application processor 30 may be provided outside the housing of the robot apparatus 20.
  • the application processor 30 can have a high processing capacity and can perform complicated arithmetic processing.
  • the sensor processor 50 does not provide the image data to the application processor 30, but the image data may be provided when the user sets a predetermined operation mode.
  • the event-driven sensor 56, the motion sensor 72, and the touch sensor 74 may be driven by standby power while the main power of the robot device 20 is off.
  • the main power supply may be automatically turned on when at least one of the event-driven sensor 56, the motion sensor 72, and the touch sensor 74 detects some information.
  • the present invention can be used for a system that analyzes image data.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Multimedia (AREA)
  • Manipulator (AREA)
  • Image Analysis (AREA)
PCT/JP2020/001715 2019-01-25 2020-01-20 画像解析システム Ceased WO2020153298A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/423,935 US12103162B2 (en) 2019-01-25 2020-01-20 Robotic device having an image analysis system
CN202080009904.4A CN113316505B (zh) 2019-01-25 2020-01-20 图像分析系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-011646 2019-01-25
JP2019011646A JP7190919B2 (ja) 2019-01-25 2019-01-25 画像解析システム

Publications (1)

Publication Number Publication Date
WO2020153298A1 true WO2020153298A1 (ja) 2020-07-30

Family

ID=71735584

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/001715 Ceased WO2020153298A1 (ja) 2019-01-25 2020-01-20 画像解析システム

Country Status (4)

Country Link
US (1) US12103162B2 (enExample)
JP (1) JP7190919B2 (enExample)
CN (1) CN113316505B (enExample)
WO (1) WO2020153298A1 (enExample)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117999578A (zh) * 2021-09-24 2024-05-07 索尼集团公司 信息处理系统、信息处理方法和程序

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08202679A (ja) * 1995-01-23 1996-08-09 Sony Corp ロボット
JP2018529175A (ja) * 2015-08-28 2018-10-04 インペリアル・カレッジ・オブ・サイエンス・テクノロジー・アンド・メディスン 多方向カメラを使用した空間のマッピング

Family Cites Families (76)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4163183A (en) 1975-10-28 1979-07-31 Unimation, Inc. Programmable automatic assembly system
US4370721A (en) 1980-09-02 1983-01-25 Cincinnati Milacron Inc. Method for modifying programmed positions by using a programmably controlled surface sensing element
JPS5822411A (ja) 1981-08-04 1983-02-09 Fanuc Ltd 工業用ロボツト制御方式
JPS62232006A (ja) 1986-04-02 1987-10-12 Yokogawa Electric Corp ロボツト・システム
US5579444A (en) 1987-08-28 1996-11-26 Axiom Bildverarbeitungssysteme Gmbh Adaptive vision-based controller
US4865791A (en) 1988-02-03 1989-09-12 The Excello Specialty Company Method of forming a surface protector with an expansible pocket
US4969108A (en) 1988-04-08 1990-11-06 Cincinnati Milacron Inc. Vision seam tracking method and apparatus for a manipulator
US5319443A (en) 1991-03-07 1994-06-07 Fanuc Ltd Detected position correcting method
JPH08150581A (ja) 1994-11-25 1996-06-11 Yaskawa Electric Corp ロボット制御装置
US6610917B2 (en) * 1998-05-15 2003-08-26 Lester F. Ludwig Activity indication, external source, and processing loop provisions for driven vibrating-element environments
JP2001322079A (ja) 2000-05-15 2001-11-20 Sony Corp 脚式移動ロボット及びその動作教示方法
US6442451B1 (en) 2000-12-28 2002-08-27 Robotic Workspace Technologies, Inc. Versatile robot control system
JP2002239952A (ja) 2001-02-21 2002-08-28 Sony Corp ロボット装置、ロボット装置の行動制御方法、プログラム及び記録媒体
US7076332B2 (en) 2002-01-18 2006-07-11 National Instruments Corporation System and method for invoking execution of a sequence of operations that includes motion control, machine vision, and data acquisition (DAQ) functionality
KR100877983B1 (ko) 2002-03-11 2009-01-12 주식회사 엘지이아이 로봇의 목표물 추종장치 및 방법
JP3910871B2 (ja) * 2002-03-28 2007-04-25 株式会社東芝 ロボット及びロボットの視野制御方法
US7076322B2 (en) 2002-03-29 2006-07-11 National Instruments Corporation System and method for satisfying move constraints when performing a motion control sequence
CN1476963A (zh) 2002-06-17 2004-02-25 ������������ʽ���� 分布式控制系统和分布式控制方法
WO2004033159A1 (ja) 2002-10-11 2004-04-22 Fujitsu Limited ロボット制御アルゴリズム構築装置、ロボット制御アルゴリズム構築プログラム、ロボット制御装置、ロボット制御プログラム、およびロボット
JP4007899B2 (ja) * 2002-11-07 2007-11-14 オリンパス株式会社 運動検出装置
JP3965686B2 (ja) * 2002-12-19 2007-08-29 株式会社日立製作所 視覚情報処理装置および適用システム
US6865441B2 (en) 2002-12-19 2005-03-08 National Instruments Corporation Parallel trajectory generation, interpolation, and control in a motion control application
US6804580B1 (en) 2003-04-03 2004-10-12 Kuka Roboter Gmbh Method and control system for controlling a plurality of robots
JP4587738B2 (ja) 2003-08-25 2010-11-24 ソニー株式会社 ロボット装置及びロボットの姿勢制御方法
JP4661074B2 (ja) 2004-04-07 2011-03-30 ソニー株式会社 情報処理システム、情報処理方法、並びにロボット装置
CN100480004C (zh) 2004-07-15 2009-04-22 上海交通大学 焊接机器人单目视觉传感器的手-眼关系快速标定方法
KR20070075957A (ko) 2006-01-17 2007-07-24 주식회사 로보스타 멀티 타스킹을 위한 타스크 기반 로봇 제어시스템
FR2896441B1 (fr) 2006-01-23 2009-07-03 Jerome Grosbois Procede et systeme permettant la prehension automatisee de piece(s)
JP2008183690A (ja) 2007-01-31 2008-08-14 Nachi Fujikoshi Corp ロボット制御装置及びロボット制御システム
AU2008353981B2 (en) 2008-04-02 2013-01-31 Irobot Corporation Robotics systems
JP5241626B2 (ja) 2009-06-18 2013-07-17 アイシン・エィ・ダブリュ工業株式会社 高耐熱性を有すロックアップクラッチ
JP4837116B2 (ja) 2010-03-05 2011-12-14 ファナック株式会社 視覚センサを備えたロボットシステム
JP5321532B2 (ja) 2010-04-28 2013-10-23 株式会社安川電機 ロボットキャリブレーション装置及びキャリブレーション方法
US9014848B2 (en) 2010-05-20 2015-04-21 Irobot Corporation Mobile robot system
JP5671281B2 (ja) * 2010-08-20 2015-02-18 キヤノン株式会社 位置姿勢計測装置、位置姿勢計測装置の制御方法及びプログラム
JP2013151045A (ja) 2012-01-25 2013-08-08 Toshiba Tec Corp 情報処理装置及びプログラム
EP2634670A1 (en) * 2012-03-01 2013-09-04 Asplund Data AB A data input device
JP5943676B2 (ja) 2012-03-30 2016-07-05 株式会社デンソーウェーブ ロボットの制御システム及びエンコーダシステム
CA2874101C (en) * 2012-05-30 2021-01-12 Ecole Polytechnique Federale De Lausanne (Epfl) Apparatus and method for restoring voluntary control of locomotion in neuromotor impairments
EP2994803A1 (en) 2013-05-10 2016-03-16 CNH Industrial America LLC Control architecture for multi-robot system
JP6261209B2 (ja) 2013-07-04 2018-01-17 キヤノン株式会社 複数の生産装置を備えるシステム
WO2015111261A1 (ja) 2014-01-21 2015-07-30 株式会社ソニー・コンピュータエンタテインメント 情報処理装置および情報処理方法
JP2014097572A (ja) 2014-02-28 2014-05-29 Seiko Epson Corp 把持ロボットおよび把持方法
EP3107069A4 (en) 2014-03-24 2017-10-04 Hitachi, Ltd. Object detection apparatus, object detection method, and mobile robot
US9931753B1 (en) * 2014-07-24 2018-04-03 Boston Dynamics, Inc. Methods and devices for automatic gait transition
JP6375810B2 (ja) 2014-09-16 2018-08-22 セイコーエプソン株式会社 画像処理装置、及びロボットシステム
JP6736257B2 (ja) * 2015-04-02 2020-08-05 キヤノン株式会社 情報処理装置、情報処理方法、プログラム
KR102055317B1 (ko) 2015-08-25 2020-01-22 카와사키 주코교 카부시키 카이샤 로봇시스템
US10976343B2 (en) 2015-09-09 2021-04-13 Sony Corporation Sensor device, sensor system, and information processing device
KR102235166B1 (ko) 2015-09-21 2021-04-02 주식회사 레인보우로보틱스 실시간 로봇 시스템, 로봇 시스템 제어 장치 및 로봇 시스템 제어 방법
US10173324B2 (en) 2015-11-16 2019-01-08 Abb Schweiz Ag Facilitating robot positioning
JP6604831B2 (ja) * 2015-11-30 2019-11-13 キヤノン株式会社 画像処理装置、画像処理装置の制御方法及びプログラム
JP6714393B2 (ja) 2016-03-02 2020-06-24 キヤノン株式会社 計測装置、システム、計測方法、および物品の製造方法
US10241514B2 (en) * 2016-05-11 2019-03-26 Brain Corporation Systems and methods for initializing a robot to autonomously travel a trained route
JP6514166B2 (ja) 2016-09-16 2019-05-15 ファナック株式会社 ロボットの動作プログラムを学習する機械学習装置，ロボットシステムおよび機械学習方法
RU2647670C1 (ru) * 2016-09-27 2018-03-16 Общество с ограниченной ответственностью "Аби Девелопмент" Автоматизированные способы и системы выявления на изображениях, содержащих документы, фрагментов изображений для облегчения извлечения информации из выявленных содержащих документы фрагментов изображений
JP2018085575A (ja) 2016-11-21 2018-05-31 カシオ計算機株式会社 画像処理装置、解析システム、画像処理方法及びプログラム
CN108161931A (zh) 2016-12-07 2018-06-15 广州映博智能科技有限公司 基于视觉的工件自动识别及智能抓取系统
JP2018126796A (ja) 2017-02-06 2018-08-16 セイコーエプソン株式会社 制御装置、ロボットおよびロボットシステム
US10268201B2 (en) * 2017-02-28 2019-04-23 Mitsubishi Electric Research Laboratories, Inc. Vehicle automated parking system and method
JP6880982B2 (ja) 2017-04-21 2021-06-02 セイコーエプソン株式会社 制御装置およびロボットシステム
US10621751B2 (en) 2017-06-16 2020-04-14 Seiko Epson Corporation Information processing device and computer program
WO2019073920A1 (ja) * 2017-10-12 2019-04-18 ソニー株式会社 情報処理装置、移動装置、および方法、並びにプログラム
CN108607819A (zh) 2018-04-25 2018-10-02 重庆邮电大学 物料分拣系统及方法
US10946528B2 (en) 2018-06-01 2021-03-16 Irepa International, LLC Autonomous companion mobile robot and system
JP6473917B1 (ja) 2018-06-22 2019-02-27 ソフトサーボシステムズ株式会社 モーション制御プログラム、モーション制御方法及びモーション制御装置
KR102258667B1 (ko) 2018-06-22 2021-06-01 소프트 서보 시스템즈 가부시키가이샤 모션 제어 프로그램, 모션 제어 방법 및 모션 제어 장치
EP3812855B1 (en) 2018-06-22 2024-01-17 Soft Servo Systems, Inc. Motion control program, motion control method, and motion control device
JP7309371B2 (ja) * 2019-01-25 2023-07-18 株式会社ソニー・インタラクティブエンタテインメント ロボット制御システム
JP7417356B2 (ja) * 2019-01-25 2024-01-18 株式会社ソニー・インタラクティブエンタテインメント ロボット制御システム
US12179337B2 (en) 2019-12-18 2024-12-31 Carla R. Gillett Modular robotic service vehicle
US11498211B2 (en) 2019-12-30 2022-11-15 Intrinsic Innovation Llc Composability framework for robotic control system
US11393434B2 (en) * 2020-07-09 2022-07-19 Industrial Technology Research Institute Method, processing device, and display system for information display
CN112043397B (zh) * 2020-10-08 2021-09-24 深圳市精锋医疗科技有限公司 手术机器人及其运动错误检测方法、检测装置
CN113524235B (zh) * 2021-07-15 2022-03-11 华中科技大学 一种面向动态交互的对象肢体时变刚度辨识方法及装置
US20230046365A1 (en) * 2021-08-12 2023-02-16 Hill-Rom Services, Inc. Techniques for three-dimensional analysis of spaces

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08202679A (ja) * 1995-01-23 1996-08-09 Sony Corp ロボット
JP2018529175A (ja) * 2015-08-28 2018-10-04 インペリアル・カレッジ・オブ・サイエンス・テクノロジー・アンド・メディスン 多方向カメラを使用した空間のマッピング

Also Published As

Publication number Publication date
US12103162B2 (en) 2024-10-01
JP2020119398A (ja) 2020-08-06
US20220080603A1 (en) 2022-03-17
JP7190919B2 (ja) 2022-12-16
CN113316505A (zh) 2021-08-27
CN113316505B (zh) 2024-04-30

Similar Documents

Publication Publication Date Title
JP7417356B2 (ja) ロボット制御システム
JP7309371B2 (ja) ロボット制御システム
KR100898435B1 (ko) 로봇 장치 및 그 제어 방법
JP6496396B2 (ja) 線形予測位置及び速度コントローラに基づいた全方向車輪を有する人間型ロボット
KR101942167B1 (ko) 전방향 휠을 가지고 선형 예측 위치 및 속도 제어기를 기초한 휴머노이드 로봇
US11992945B2 (en) System and methods for training robot policies in the real world
JP2004298977A (ja) 行動制御装置、行動制御方法及び行動制御プログラム、並びに移動型ロボット装置
JP2004110802A (ja) 環境同定装置、環境同定方法、プログラム及び記録媒体、並びにロボット装置
CN107127760A (zh) 一种足履复合式人形机器人
JP2003266349A (ja) 位置認識方法、その装置、そのプログラム、その記録媒体及び位置認識装置搭載型ロボット装置
JP7190919B2 (ja) 画像解析システム
JP4281286B2 (ja) ロボット装置及びその制御方法
JP5539001B2 (ja) 制御装置
Zhang et al. An interactive control system for mobile robot based on cloud services
JP4193098B2 (ja) トラッキング装置、トラッキング装置のトラッキング方法及びロボット装置
JP5539000B2 (ja) 制御装置
JP2003266352A (ja) ロボット装置及びロボット装置の制御方法
Motoc et al. Zero moment point/Inverted pendulum-based walking algorithm for the NAO robot
JP2002205290A (ja) 脚式移動ロボットの制御装置及び制御方法
Sasaki et al. Optimizing and Predicting Human-Symbiotic Robot Trajectory
Chen An Intelligent human-tracking robot based-on kinect sensor
JP2004299006A (ja) ロボット装置及びロボット装置による顔認識状況の提示方法
Ercan et al. Vision based motion planning of humanoid robots
Baizid et al. ViRAT: An advanced multi-robots platform

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20745229

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20745229

Country of ref document: EP

Kind code of ref document: A1