JP2002251615A - Device and method for processing image, robot device, and method for controlling the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and method for processing an image capable of accurately detecting a moving body in an image, and provide a robot and a method for controlling the same capable of improving an entertainment property. SOLUTION: The device and method for processing an image detects a moving quantity of a whole image on the basis of image information, and detects a movement in the image by performing a predetermined movement detecting process which considers a detection result. The robot device and the method for controlling the same detects a moving quantity of the whole image on the basis of image information outputted from an imaging means for imaging an external, detects a movement in an image by performing a predetermined movement detecting process which considers a detection result, and reflects the detection result to an action.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method, a robot apparatus and a control method therefor, and is suitably applied to, for example, a four-legged robot apparatus.

[0002]

2. Description of the Related Art In recent years, a pet robot of a four-legged walking type that performs an action in response to a command from a user or the surrounding environment has been developed by the present applicant.

[0003] Such a pet robot is a CCD (Charge).
Coupled Device) A camera and microphone are mounted, and the surrounding conditions captured by the CCD camera,
Based on the command sound from the user and surrounding sounds collected by the microphone, the surrounding situation and the presence or absence of the command from the user are determined, and the action is autonomously determined based on the determination result and expressed. It was done in.

[0004]

As a method of detecting a moving object in an image by image processing, various methods have been conventionally proposed and put to practical use. These methods are roughly divided into a first method of obtaining a pixel difference between image frames, a second method of obtaining an optical flow at an arbitrary point on an image using a spatio-temporal filter, In the third method, the matching of the small area is performed, and the motion of the small area is obtained. In this case, any of these methods is based on the premise that the camera for capturing an image is fixed and the detected motion is an external motion.

[0005] However, in a robot having a multi-degree-of-freedom joint, for example, such as the above-mentioned pet robot, the mounted CCD camera moves large and complicated with the operation of the robot.

Therefore, in such a pet robot,
When trying to detect a moving object in an image by the first to third methods as described above, it is difficult to detect the movement even if the moving object is actually imaged because the entire image greatly moves between frames. there were. On the contrary, there is also a problem that a moving object is erroneously detected by its own operation even if there is no movement around.

If the moving object cannot be accurately detected as described above, the pet robot may behave as if the moving object is present even though the moving object does not actually exist around the pet robot. However, there is a problem in that the expressed behavior becomes unnatural, such as the appearance of a behavior as if there is no moving body despite the presence of a moving body in the vicinity.

Therefore, in such a pet robot, if it is possible to accurately detect a moving object existing in the surroundings regardless of the presence or absence of its own movement, it is possible to prevent the occurrence of unnatural behavior. Therefore, it is thought that the entertainment property can be further improved.

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and proposes an image processing apparatus and method capable of accurately detecting a moving object in an image, a robot apparatus capable of improving entertainment characteristics, and a control method thereof. Is what you do.

[0010]

According to the present invention, there is provided an image processing apparatus comprising: first detecting means for detecting a movement amount of an entire image based on image information; Second detection means for detecting a motion in an image by a predetermined motion detection process in consideration of the detection result is provided. As a result, the image processing apparatus can accurately detect a moving object in an image regardless of whether or not the entire image has moved.

Further, in the present invention, in the image processing method, a first step of detecting a moving amount of the entire image based on image information and a predetermined motion detecting process in consideration of the detected moving amount are performed. And a second step of detecting As a result, according to this image processing method, it is possible to accurately detect a moving object in an image regardless of whether or not the entire image is moving.

Further, according to the present invention, in a robot apparatus which autonomously determines and expresses an action in a robot apparatus, a movement amount of an entire image based on image information output from an image pickup means for picking up an external image is detected. A first detecting means and a second detecting means for detecting a motion in an image by a predetermined motion detecting process in consideration of the detection result of the first detecting means, and the detection result of the second detecting means is used as an action. Was reflected in As a result, the robot device accurately detects a moving object in an image based on image information output from the imaging unit and reflects the detection result regardless of whether or not the imaging unit moves along with its own operation. You can take action.

Further, according to the present invention, in the control method of the robot apparatus, the imaging means mounted on the robot captures an image of the outside and detects a moving amount of the entire image based on image information output from the imaging means. A first step, a second step of detecting a motion in the image by a predetermined motion detection process in consideration of the detected movement amount,
And a third step of reflecting the detection result in the second step to the behavior of the robot device. As a result, according to the control method of the robot device, regardless of whether or not the imaging device moves according to the operation of the robot device, the moving object in the image based on the image information output from the imaging device is accurately detected, The action reflecting the detection result can be performed by the robot device.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below in detail with reference to the drawings.

(1) Configuration of Pet Robot According to the Present Embodiment In FIG. 1, reference numeral 1 denotes a pet robot according to the present embodiment as a whole, and leg units 3A to 3D are connected to the front, rear, left and right of body unit 2, respectively. The head unit 4 and the tail unit 5 are connected to the front end and the rear end of the body unit 2, respectively.

As shown in FIG. 2, a CPU (Central Processing Unit) 10 and a DRA
M (Dynamic Random Access Memory) 11, Flash ROM (Read Only Memory) 12, PC (Personal Com
puter) A control unit 16 formed by connecting a card interface 13 and a signal processing circuit 14 to each other via an internal bus 15 and a battery 17 as a power source of the pet robot 1 are housed therein. The body unit 2 has an angular acceleration sensor 18 for detecting the acceleration of the direction and the movement of the pet robot 1.
And an acceleration sensor 19 are also stored.

The head unit 4 includes a CCD camera 20 for capturing an image of an external situation, and a touch sensor for detecting a pressure applied by a physical action such as “stroke” or “hit” from the user. 21 and a distance sensor 22 for measuring the distance to an object located ahead
, A microphone 23 for collecting external sounds, a speaker 24 for outputting a sound such as a cry, and an LED (not shown) corresponding to an external “eye” of the pet robot 1
(Light Emitting Diode) and the like are respectively arranged at predetermined positions.

Further, joints of the leg units 3A to 3D, connecting portions of the leg units 3A to 3D and the body unit 2, connecting portions of the head unit 4 and the body unit 2, and a tail unit 5 Actuators 25 (25 ₁ , 25 ₂ , 25 ₃ ...) And potentiometers 26 (26 ₁ , 26 ₂ , 26 ₃ ...) Are provided at the base of the tail 5A (FIG. 1). Have been.

The angular velocity sensor 18, the acceleration sensor 19, the touch sensor 21, the distance sensor 22, the microphone 23, the speaker 24, the LED, each actuator 25 and each potentiometer 26 are respectively connected to a hub 27.
They are connected to the signal processing circuit 14 of the control unit 16 in a tree shape via (27 _{1 to} 27 _n ). The CCD camera 20 and the battery 17 are respectively connected to the signal processing circuit 1
4 is directly connected.

At this time, the signal processing circuit 14 includes sensor data supplied from various sensors such as an angular velocity sensor 18, an acceleration sensor 19, a touch sensor 21, a distance sensor 22, and potentiometers 26 via a hub 27, and a CCD camera. The image data supplied from the microphone 20 and the audio data supplied from the microphone 23 are sequentially fetched, and these are respectively transferred to the DRAM 1 via the internal bus 15.
1 is sequentially stored in a predetermined position. The signal processing circuit 14
Sequentially captures remaining battery power data indicating the remaining battery power supplied from the battery 17 and
Is stored in a predetermined position.

The sensor data, image data, voice data, and remaining battery data stored in the DRAM 11 are used when the CPU 10 controls the operation of the pet robot 1 thereafter.

In practice, when the power of the pet robot 1 is turned on, the CPU 10 executes the control program stored in the memory card 28 or the flash ROM 12 inserted in the PC card slot (not shown) of the body unit 2 into the PC card interface. 13, or directly read out, and develops the data in the DRAM 11.

Further, the CPU 10 thereafter, based on the various sensor data, image data, audio data, and remaining battery power data sequentially stored in the DRAM 11 from the signal processing circuit 14 as described above, and from the user, Judge whether or not there is an instruction for and action.

Further, the CPU 10 determines the result of this determination and D
The next action is determined based on the control program developed in the RAM 11, and the first drive signal corresponding to the determined result is generated and transmitted to the necessary actuator 25, so that the head unit 4 can be moved up, down, left and right. , The tail unit 5A of the tail unit 5 can be moved, and the leg units 3A to 3D are driven to walk.

At this time, the CPU 10 generates an audio signal and a second drive signal as necessary, and supplies them to the speaker 24 and the LED of the “eye” through the signal processing circuit 14 so that the audio signal is generated. Output the sound based on the second, or turn on the LED based on the second drive signal,
Turn off or blink.

As described above, the pet robot 1 can behave autonomously in response to the situation of itself and the surroundings, instructions and actions from the user, and the like.

FIG. 3 shows a specific configuration of the signal processing circuit 14. As is clear from FIG. 3, the signal processing circuit 1
4 is a DMA (Direct Memory Access) controller 3
0, DSP (Digital Signal Processor) 31, peripheral interface 32, timer 33, FBK / C
DT (Filter Bank / Color Detection) 34, IPE
(Inner Product Engin) 35, a serial bus host controller 36, and a serial bus 37 are sequentially connected to a bus 40 via a bus 38 and a bus arbiter 39 for arbitrating the right to use the bus 38, and the bus 40 is a DRAM. It is connected to the DRAM 11 (FIG. 2), the CPU 10 (FIG. 2), and the flash ROM 12 (FIG. 2) via the interface 41, the host interface 42, and the ROM interface 43, and the parallel interface 44, the battery manager 45, and the serial port are connected to the peripheral interface 32. 46 is connected.

In this case, the angular velocity sensor described with reference to FIG.
Sensor 18, acceleration sensor 19, touch sensor 21, distance sensor
Sensor 22, microphone 23, speaker 24, each activator
Devices such as a heater 25 and each potentiometer 26
Is the hub 27 (27 ₁~ 27_n) Through Syria
Connected to the host controller 36 and a CCD
The camera 20 (FIG. 2) is connected to the FBK / CDT 34,
The battery 17 (FIG. 2) is connected to the battery manager 45.
It is connected.

Then, the serial host controller 36
Is an angular velocity sensor 18 among the connected devices,
Acceleration sensor 19, touch sensor 21, distance sensor 22
And a DMA controller 3 functioning as a bus master for transferring data by sequentially taking in sensor data given from each sensor such as each potentiometer 26 and the like.
Under the control of 0, these sensor data are sequentially provided to the DRAM 11 via the bus 38, the bus arbiter 39, the bus 40 and the DRAM interface 41 and stored therein.

Further, the serial host controller 36
The voice data given from the microphone 23 is converted to DSP3
1, the DSP 31 performs predetermined data processing on the supplied audio data supplied from the host controller 36, and outputs the resulting audio data under the control of the DMA controller 30 under the control of the DMA controller 30. The data is sequentially transferred to the DRAM 11 via the bus 38, the bus arbiter 39, the bus 40, and the DRAM interface 41, and is stored in a predetermined storage area in the DRAM 11.

The FBK / CDT 34 divides the image data supplied from the CCD camera 20 into a plurality of resolutions while performing color recognition, and fetches the obtained image data.
Under the control of the DMA controller 30, the data is transferred to the DRAM 11 (FIG. 2) via the bus 38, the bus arbiter 39, the bus 40, and the DRAM interface 41 in order, and is transferred to a designated storage area in the DRAM 11 as described later. To be stored.

Further, the battery manager 45 stores the remaining battery data indicating the remaining energy notified from the battery 17 under the control of the DMA controller 30 into the peripheral interface 32, the bus 38, the bus arbiter 39, the bus 40, and the DRAM. Interface 41
Are sequentially transferred to the DRAM 11 via the
It is stored in a predetermined storage area in M11.

On the other hand, the signal processing circuit 14 drives the first drive signal for driving each actuator 25 provided from the CPU 10 (FIG. 2) via the bus 15 as described above, the audio signal, and the LED. Is input via the host interface 42.

The signal processing circuit 14 sends these signals to the bus 40, the bus arbiter 39, the bus 38, the serial bus host controller 36, and the corresponding hub 27 (27 _1).
... 27 _n ) (FIG. 2) sequentially to the corresponding actuators 25 (25 ₁ , 25 ₂ , 25 ₃ ...) (FIG. 2), the speaker 24 or the LED.

As described above, in the signal processing circuit 14, each sensor, the CCD camera 20, the microphone 23,
Each device such as a speaker and each actuator 25;
The CPU 10 can perform various signal processing necessary for controlling the behavior of the pet robot 1 with the CPU 10.

The IPE 35 is a two-dimensional digital filter capable of performing a matching process between images at a high speed. As will be described later, the two images are shifted one pixel at a time, and the correlation between the two images at each shift position is determined. Can be sequentially calculated based on a predetermined arithmetic expression.

The serial bus 37 is an interface for connecting to a remote computer such as a personal computer (PC).
The serial port 46 is a connector for connecting to the remote computer. Furthermore, timer 33
Plays a role as a clock inside the pet robot 1.

(2) Method of Transferring Image Data in Pet Robot 1 Here, a method of transferring image data input from the CCD camera 20 (FIG. 2) to the DRAM 11 (FIG. 2) will be described. As shown in FIG. 4, the image data output from the CCD camera 20 is transmitted to the signal processing circuit 14 (FIG. 2).
Of the FBK / CDT 34 of the
After the predetermined image processing is performed at 34, the bus arbiter 3 under the control of the DMA controller 30 (FIG. 3).
9 (FIG. 3) and DRAM interface 41 (FIG. 3)
Are sequentially transferred to the DRAM 11.

The DMA controller 30 reads out the DMA list generated by the CPU 10 (FIG. 2) from the DRAM 11 via the bus arbiter 39 and the DRAM interface 41 in order, and transfers the image data described in the DMA list or the transfer destination. The image data is transferred on the basis of the information on the image data.

Therefore, in this pet robot 1,
By rewriting the DMA list by the CPU 10,
The DMA controller 30 can transfer image data output from the DBK / CDT 34 to an arbitrary position on the DRAM 11. Thus, in the pet robot 1,
By transferring the image data to the DRAM 11 and developing it, high-speed data processing can be performed by effectively using the cache function of the CPU 10.

Hereinafter, a method of transferring image data in the pet robot 1 will be specifically described with reference to FIG. 4 showing the configuration of the FBK / CDT 34.

The image data output from the CCD camera 20 is composed of 8-bit parallel data, a clock signal, and a synchronization signal. The image data is input to the input image interface 50 of the FBK / CDT 34 and converted into a predetermined format. It is supplied to the calculation unit 51.

The CPU 10 determines various filter coefficients based on the control program stored in the DRAM 11 and gives the determined filter coefficients to the parameter storage unit 52 to store them.

The filter operation section 51 performs a color filter operation on the image data provided from the input image interface 50 using the filter coefficients stored in the parameter storage section 52, thereby performing color recognition of the image data. While generating image data of a plurality of resolutions.

At the subsequent stage of the filter operation unit 51, two buffers 53A and 53B each having a storage capacity capable of storing one line of image data are provided. The data is sent to the buffers 53A and 53B alternately for each line and stored therein.

At this time, the DMA interface 54
Buffer 5 of one of the three buffers 53A, 53B
While one line of image data is stored in 3A and 53B, one line of image data is read from the other buffers 53A and 53B, and the read one line of image data is read based on an instruction from the DMA controller 30. D
By transferring the image data to the RAM 11, a plurality of image data are sequentially transferred to the DRAM 11. The buffers 53A, 5A
3B has only a storage capacity for one line, so the storage capacity is small and the FBK / CDT having a simple configuration is used.
34 are realized.

By the way, such transfer of image data is performed as follows.
This is performed by the DMA controller 30 functioning as a bus master. That is, the DMA interface 54
When one of the two buffers 53A and 53B stores one line of image data, it sends a transfer request signal called a DMA request to the DMA controller 30.

Then, the DMA controller 30
When the request is supplied, first, the right to use the bus 38 (FIG. 3) is obtained, and then the corresponding DMA list is read from the DRAM 11 based on the address notified by the CPU 10.

Then, the DMA controller 30
DRA which is the transfer destination of the image data based on the MA list
Addresses on M11 are sequentially generated, and a write signal is issued to the bus 38. The FBK / CDT 34 synchronizes with the write signal to the bus 38 and
The image data stored in 3A and 53B is read out and sequentially transmitted to the bus 38.

The image data output from the FBK / CDT 34 is written to the DRAM 11 at the same time as the image data is output from the FBK / CDT 34, so that the transfer of the image data is performed at a very high speed. Incidentally, since the setting regarding the transfer destination of the image data is performed by the DMA list, the DMA controller 30 stores the image data in the DR.
The data can be transferred to any position on the AM 11.

When the transfer of the image data for one line stored in the buffers 53A and 53B is completed, the FBK / CDT 34 cancels the DMA request to the DMA controller 30 and terminates the transfer of the image data. In response, the DMA controller 30
Information such as the amount of image data transferred by the time the DMA request is withdrawn is written to the DMA list, and then the bus 38 is released. And DMA
When the DMA request is supplied again, the controller 30 reads the above-described DMA list from the DRAM 11 and performs the transfer following the previous one.

As described above, the pet robot 1 is capable of transferring image data at high speed.

(3) Image Processing Method in Pet Robot 1 Next, an image processing method in the pet robot 1 will be described. At this time, the software configuration of the control program of the pet robot 1 will be described first.

The pet robot 1 has an operating system having the following functions. 1. 1. Can execute multiple tasks. 2. Data can be transferred between tasks. Memory (DRAM 11) can be shared between tasks

Specifically, Aperios (trademark of Sony Corporation) is used as an operating system.
Is used as a communication method between each object on this operating system, for example, Japanese Patent Application No.
-006491 is used.

FIG. 5 shows a software configuration of a control program in the pet robot 1. In FIG. 5, the device driver layer 60 includes:
A device driver set 6 located at the lowest layer of the control program and including a plurality of device drivers
1.

In this case, each device driver is a CCD.
An object that is allowed to directly access hardware used in a normal computer, such as the camera 20 (FIG. 2) and the timer 33 (FIG. 3), and performs processing upon receiving an interrupt from the corresponding hardware.

The robotic server object 62 is located on the upper layer of the device driver layer 60, and includes, for example, the aforementioned sensors and actuators 25 (2
5 ₁ , 25 ₂ , 25 ₃ ...) (FIG. 2) and the like, and a virtual robot 63 which is a software group for providing an interface for accessing hardware, and a power which is a software group for managing switching of power supply and the like. It comprises a manager 64, a device driver manager 65 which is a group of software for managing various other device drivers, and a designed robot 66 which is a group of software for managing the mechanism of the pet robot 1. I have.

The manager object 67 includes an object manager 68 and a service manager 69
It is composed of In this case, the object manager 68 has the robotic server object 62,
A software group that manages activation and termination of each software group included in the middleware layer 70 and the application layer 71.
Reference numeral 9 denotes a software group that manages the connection of each object based on the connection information between the objects described in the connection file stored in the memory card 28 (FIG. 2).

The middleware layer 70 is located on the upper layer of the robotic server object 62 and is composed of a group of software for providing basic functions of the pet robot 1 such as image processing and sound processing. I have.
The application layer 71 is located above the middleware layer 70, and
It is composed of a software group for determining the behavior of the pet robot 1 based on the processing result processed by each software group constituting the layer 70.

The manager object 67 is composed of an object manager and a service manager. Among them, the object manager is a software group constituting a robotic server object middleware layer and an application layer. The service manager is a software group that manages the connection of each object based on the connection information between the objects described in the connection file stored in the memory card. .

The image processing performed inside the pet robot 1 based on the control program having such a configuration will be described below by taking as an example a case where a plurality of image processing tasks such as color detection, motion detection, and obstacle detection are simultaneously executed. This will be described with reference to FIG. 8 using FIGS. 6 and 7.

FIGS. 6 and 7 show a procedure from the start of the image processing (hereinafter referred to as an image processing procedure RT1).
When the power of the pet robot 1 is turned on, the image processing procedure RT1 is started, and in the subsequent step SP2, the control unit 16 (FIG. 2)
The FBK / CDT driver 61 shown in FIG. 8 in which the CPU 10 constitutes a device driver set 61 (FIG. 5)
A is started to initialize the FBK / CDT 34.

In the following step SP3, the CPU 10
Is the robotic server object 62 (FIG. 5)
Start At this time, the virtual robot 63 (FIG. 5) is also activated. And the virtual robot 63
After the start-up, as shown in FIG. 8, a plurality of shared memories 80 for storing image data are generated on the DRAM 11 (FIG. 2). Set information about attributes.

When the activation of the robotic server object 62 (FIG. 5) is completed, the CPU 10 sequentially activates the image processing objects constituting the middleware layer 70 (FIG. 5). Incidentally, the storage capacity of the generated shared memory 80 is the virtual robot 63.
It can be easily changed according to the instruction in FIG.

In the following step SP4, the virtual robot 63 notifies the FBK / CDT driver 61A of the address of the desired shared memory 80 among the plurality of shared memories 80 generated on the DRAM 11, and shares the notified address. It issues a read request to transfer image data to the memory 80 to the FBK / CDT driver 61A.

The FBK / CDT driver 61A
Upon receiving the read request, in the following step SP5, a DMA list 81 for transferring image data to the designated shared memory 80 is generated and stored in the DRAM 11 (FIG. 2). At that time, FBK / CDT driver 61A
After notifying the address of the DMA list 81 stored in the DRAM 11 to the DMA controller 30 (FIG. 3),
Output of image data to the FBK / CDT 34 is permitted.

In the following step SP6, FBK / C
The DT 34 receives the output permission of the image data from the FBK / CDT driver 61A, and transfers the image data for one field to the designated shared memory 80.
An interrupt is generated for the DT driver 61A, and software called an interrupt handler is started. Then, in step SP7, the interrupt handler notifies the virtual robot 63 that the transfer of the image data has been completed.

At this time, in the following step SP8, the virtual robot 63 determines whether or not the required number of fields of image data have been transferred. If the virtual robot 63 obtains a negative result in step SP8, the process proceeds to step SP9, where the address of the shared memory 80 to be transferred next is set to the FBK / CDT driver 6.
After issuing the read request by passing the request to 1A, step SP5
And then return to steps SP5-SP6-SP7-SP8-SP9 until a positive result is obtained in step SP8.
-Repeat the loop of SP5.

On the other hand, if the CPU 10 obtains an affirmative result in step SP8, it proceeds to step SP10 and waits for the activation of each image processing object constituting the middleware layer 70 (FIG. 5). Is completed, the process proceeds to step SP11.

The service manager 69 of the manager object 67 has the middleware layer 7
When the activation of each image processing object is completed, in step SP11, the image processing objects of the connection destinations are notified to the image processing objects, and a communication path is opened between the image processing objects.

For example, the service manager 69 is shown in FIG.
As is apparent from the above, the image processing object 82 for color detection and the image processing object 84 for edge detection are connected to the virtual robot 63. Further, the service manager 69 includes an image processing object 85 for calculating the center of gravity.
Is connected to the image processing object 82 for color detection, and the image processing object 86 for calculating the center of gravity is connected to the image processing object 83 for motion detection.

Further, the service manager 69 hierarchically connects the obstacle detection image processing object 87 and the coordinate conversion image processing object 88 to the edge detection image processing object 84. Incidentally, the connection between such image processing objects can be changed, and various processes can be performed by changing the connection relationship between the image processing objects.

In step SP12, when the connection between the image processing objects is completed, the service manager 69 sends a start signal to each image processing object. In response to this, in the following step SP13, each image processing object sends a data request signal to the lower-layer connected image processing object based on the connection information described in the connection file.

By the way, the image processing object 8 for color detection
2. The image processing object 83 for motion detection and the image processing object 84 for edge detection are the FBK / CDT 34
The image data sequentially supplied from is read at different feed intervals, and
The robot 63 grasps the image data to be read by each image processing object among the supplied image data.

Therefore, when the virtual robot 63 receives a data request signal from each image processing object in the subsequent step SP14, the virtual robot 63 is assigned to the shared memory 80 in which the image data to be read by each image processing object is stored. ID (Identifica
) (that is, address information) is notified to each image processing object. At that time, the virtual robot 63 uses the internal reference count counter to determine the ID of each of the image processing objects.
The number of the notified image processing objects is counted, and the count value is stored.

At this time, the virtual robot 63 determines that the count value obtained by the reference count counter is not “0”.
In the shared memory 80 of D, the count number is set to “0” without overwriting the image data supplied from the FBK / CDT 34.
By overwriting and sequentially storing the image data in the shared memory 80, it is possible to avoid erroneous erasure of the image data that may be read by each image processing object.

Subsequently, in step SP15, each image processing object reads out image data from the shared memory 80 to which the notified ID is assigned. At this time, each image processing object may read image data stored in the same shared memory 80. However, since the shared memory 80 is a read-only memory, each image processing object can read image data without interfering with each other.

Next, in step SP16, each image processing object determines whether or not the required number of image data has been read out. If it is determined that the required number of image data has not been read out, the process proceeds to step SP17. Proceed to.

Each image processing object sends a data request signal for requesting the next image data to the virtual robot 63 based on the connection information described in the connection file. Then, in the following step SP18, the virtual robot 63
When the data request signal is received, the count number of the participation count measurement counter stored corresponding to the ID notified to the image processing object that has transmitted the data request signal is decremented every time the data request signal is supplied.

Then, when the count number becomes “0”, the virtual robot 63 sets the shared memory 8 to which the ID whose count has become “0” is assigned.
The protection of the image data stored in 0 is released, and the process returns to step SP14 to repeat the same operation.

Further, when each image processing object eventually obtains a positive result in step SP16, the process proceeds to step SP19, where each image processing object reads out image data from the desired shared memory 80. Apply predetermined image processing,
The processing result is sent to the upper layer object.

For example, the image processing object 82 for color detection
Detects the color from the read image data, sends the processing result to the image processing object 85 for calculating the center of gravity, and calculates the position of the detected color. Further, the motion detection image processing object 83 detects a motion area from the image data, and sends the processing result to the center of gravity calculation image processing object 86 to calculate the position and size of the detected motion area.

Further, the edge detection image processing object 84 detects an edge from the image data and sends the processing result to the obstacle detection image processing object 87 to calculate the position of the obstacle. The coordinates of the position of the obstacle in the image processing object 88 are converted.
Then, the CPU 10 thereafter proceeds to step SP20 and ends this processing (image processing procedure RT1).

As described above, each image processing object has a D
Since the image data is read from the shared memory 80 generated on the RAM 11, a general-purpose image processing object independent of the CCD camera 20 can be realized. , Various image processing can be easily performed.

(4) Moving Body Detection Algorithm in Pet Robot 1 Next, a moving body detection method in the pet robot 1 will be described.

In this pet robot 1, the image data of the image of the immediately preceding frame (hereinafter, referred to as a previous frame image) 90 shown in FIG. 9A and the current frame image shown in FIG. Based on image data of an image (hereinafter referred to as a current frame image) 91, the current frame image 91
To detect an image area 91A having a high correlation with the central portion 90A of the previous frame image 90 in the
The movement amount of the entire image (hereinafter, referred to as the movement amount of the entire image) due to the above operation is detected, and only the moving part in the image as shown in FIG. By extracting the moving object, the moving object can be accurately detected.

In this case, such a series of processing is performed as shown in FIG.
The image processing object 83 for motion detection (FIG. 8) and the image processing object 8 for calculating the center of gravity of the middle wear layer 70 (FIG. 5) according to the moving object detection processing procedure RT2 shown in FIG.
6 (FIG. 8) and the IPE3 in the signal processing circuit 14 (FIG. 2).
5 (FIG. 3). Hereinafter, a moving object detection method in such a pet robot 1 will be described.

The moving object detection processing procedure RT2 is executed by starting the image processing object 83 for motion detection in step SP30.
Started at Then, in the following step SP31, the image processing object 83 for motion detection
The image data is fetched from the designated shared memory 80 on M11 and stored, and the next step SP3
In 2, it is determined whether or not the image data is fetched for the first time.

Then, the image processing object 8 for motion detection
3 returns to step SP31 again when a positive result is obtained in step SP32, and thereafter returns to step SP31.
And step SP32 are repeated.

Further, the image processing object 8 for motion detection
When a positive result is obtained in step SP32, the process proceeds to step SP33, and as shown in FIG. 11A, the central region of the previous frame image 92 (the previous frame image 9
(Area of image area excluding predetermined upper, lower, left and right width from 2) 92A
As a template, called the central region 92A of the upper, lower, left and right to 4 divided upper left point, the upper right, lower left and each divided area in the lower right (hereinafter, these first to fourth template divided regions 92A ₁ ～92A ₄ ) Image data in order
Transfer to 35.

The motion detection image processing object 83
In the following step SP34, as shown in FIG. 11C, the entire current frame image 93 is set as a search image.
Each of upper left, upper right, lower left and lower right divided regions when the current frame image 93 is divided into four in the up, down, left and right directions while overlapping the upper or lower and left or right by a predetermined number of pixels (hereinafter referred to as a first region). ~ Fourth search image division area 93A
Transfers image data is referred to as _{1 ~93A 4)} to IPE35.

Therefore, in this example, the upper left image area (first search image division area 93A ₁ ) of the current frame image 93 delimited by the second horizontal line l _x2 and the second vertical line _ly 2 in FIG. 11C. and), and the upper right of the image region of the current frame image 93 (the second search image segmentation region 93A ₂₎ which is delimited by the second horizontal line l _x2 and the first vertical line l _y1, first horizontal line l _x1 and the The lower left image area (third search image divided area 93A ₃ ) of the current frame image 93 separated by two vertical lines l _y2 and the current frame separated by the first horizontal line l _x1 and the first vertical line l _y1 The image data of each image area of the image area at the lower right of the image 93 (the fourth search image division area 93A ₄ ) is transferred to the IPE 35.

In the following step SP35, IPE3
5 includes a front first to fourth template divided regions 92A ₁ ～92A ₄ based on the central region 92A of the frame image 92,
Performing a matching process between the search image segmentation region 93A ₁ ~93A ₄ corresponding position first to fourth of the current frame image 93.

Specifically, the IPE 35 includes a first template divided area 92A ₁ and a first search image divided area 93A _1.
Are shifted in units of one pixel in the vertical and horizontal directions, and the following equation at the shifted position (hereinafter referred to as a shifted position) is obtained.

[0096]

(Equation 1)

The matching scores given by are calculated sequentially, and the matching scores thus obtained are converted to DR.
DMA transfer is performed sequentially to the AM 11 to store it.

The IPE 35 operates similarly to the second
Search image segmentation region 93A ₂ of the template divided regions 92A ₂ and the second search of the third template divided regions 92A ₃ and the third search image divided regions 93A _3, and the fourth template divided regions 92A ₄ and the fourth DRAM for image division areas 93A _4, while shifting them by one pixel in the vertical or horizontal direction, respectively, with successively calculating a matching score given by equation (1) at each shifting position, thus resulting matching score
The data is sequentially DMA-transferred to 11 and stored.

Then, the image processing object 8 for motion detection
3, when all the matching score values are calculated, the following equation is obtained.

[0100]

(Equation 2)

By executing the operation given by
First to fourth template divided regions 92A ₁ ~92A _{4 4} single score obtained by adding the matching score for at each shift position (hereinafter, referred to as total matching score) is calculated.

Here, the image area of the current frame image 93 having the smallest value of the total matching score calculated as described above is the central area 92A of the previous frame image 92.
Therefore, the distance from the center position of the center region 92A of the previous frame image 92 to the center position of this image region is the highest in the area of the entire image between the previous frame and the current frame accompanying the movement of the pet robot 1. The amount of movement.

Therefore, the motion detection image processing object 8
3 is a step SP3 in the following step SP36.
As shown in FIG. 12, based on the respective addition results obtained in step 5, the image area of the current frame image 93 having the same overall matching score as the central area 92A of the previous frame image 92 having the smallest value ( That is, the center position (x _C2 , y _C2 ) of the region 93B having the highest correlation and surrounded by the broken line in FIG. 12 is stored, and the current frame image is _calculated from the center position (x _C2 , y _C2 ). 9
By calculating the distance and direction to the center position (x _C1 , y _C1 ) of _No. 3, the movement amount of the entire image between the previous frame and the current frame is detected.

In the following step SP37, the motion detection image processing object 83 determines whether or not the total matching score for the image area 93B detected in step SP36 is smaller than a predetermined threshold. , Its image area 93B,
It is determined whether the correlation with the central area 92A of the previous frame image 92 is lower than a predetermined level.

Here, obtaining a negative result in step SP37 means that no motion or change in luminance has occurred or has occurred in the image area 93B of the current frame image 93 between the previous frame and the current frame. This means that the movement or the change in luminance is small, and at this time, the motion detection image processing object 83
Go to 41.

On the other hand, obtaining a positive result in step SP37 means that a certain degree of movement or luminance change has occurred in the image area 93B of the current frame image 93 between the previous frame and the current frame. At this time, the image processing object 83 for motion detection
Proceeding to step SP38, the luminance level of each pixel in the central region 92A of the previous frame image 92 is calculated using the movement amount of the entire image detected in step SP36.
The difference from the luminance level of the corresponding pixel in the image area 93B of the current frame image 93 is sequentially calculated.

As a result, the central region 92A of the previous frame image 92 and the current frame image 93
Is obtained, and this data (hereinafter referred to as difference data) is output to the center-of-gravity calculation image processing object 86 (FIG. 8).

The image processing object 86 for calculating the center of gravity is
In a succeeding step SP39, based on the difference data given from the motion detection image processing object 83,
In addition to detecting an area where pixels whose difference data values are larger than a predetermined threshold set in advance, that is, an area where a motion has occurred in the image (hereinafter referred to as a motion area), The number of pixels is counted, and the position of the center of gravity of the moving area is detected.

Then, the image processing object 8 for calculating the center of gravity
6, the image area 93B of the current frame image 93 detected in the above-mentioned step SP36 in the subsequent step SP40 is located at the same position as the position detected in the previous process in which the previous frame image 92 was set as the current frame image. It is determined whether or not.

Here, the fact that the image area 93B is at the same position means that the motion of the entire image is small between the previous frame and the current frame. At this time, the image processing object 86 for calculating the center of gravity is Is compared with a preset first threshold value TH _ST . On the other hand, the fact that the image area 93B is not located at the same position means that the motion of the entire image is large between the previous frame and the current frame. Is compared with a preset second threshold TH _MV (> TH _ST ). Thus, the image processing object 86 for calculating the center of gravity
Means that the number of pixels in such a motion area is the first or second threshold value TH
_It is determined whether or not there is a motion in the image by determining whether or not _ST is larger than TH _MV .

Since the thresholds to be compared are divided according to the position of the image area 93B of the current frame image 93 detected in step SP36, the first threshold T
Pet robot 1 by setting H _ST smaller
When there is no movement, small movements can be detected sensitively, and by setting the second threshold value TH _MV to a large value, when the pet robot has movements, the sensitivity can be lowered to reduce false detection of movements. .

At the following step SP41, the CPU 1
A value of 0 determines whether or not the result of the determination of the presence or absence of the motion in step SP40 has been acquired for 10 frames. When a negative result is obtained, the process returns to step SP31, and thereafter, in step SP41, a positive result is obtained in step SP31.
Step SP41 is repeated.

When a positive result is obtained in step SP41, the image processing object 86 for calculating the center of gravity is obtained.
After resetting the count value of the number of frames, the process proceeds to step SP42, and it is determined whether or not there is actually any external movement based on the determination result of the presence or absence of the motion for the past 10 frames in step SP40. Specifically, the image processing object 86 for calculating the center of gravity determines, for example, that there is a real movement when the movement is detected in more than half of the 10 frames, and more than half of the movement is detected. If no motion is detected in the frame, it is determined that there is no motion.

Then, the image processing object 8 for calculating the center of gravity
6, when it is determined that there is truly movement, the latest frame image (the last 10
After notifying the corresponding module in the application layer 71 of the position of the center of gravity of the motion area in the last frame of the frame), the process returns to step SP31. Thereafter, the flow returns to step SP31.

In this way, the pet robot 1 can accurately detect a moving object in an image while taking into account the movement of the entire image accompanying its own movement.

(5) Reflecting the Moving Object Detection Result on Behavior Next, how the moving object detection result obtained as described above is actually reflected on the behavior of the pet robot 1 will be described.

If it is determined in step SP42 of the moving object detection processing procedure RT2 (FIG. 10) that there is a true movement, the determination result is notified to the action determination module 100 in the application layer 71 shown in FIG.

This action determination module 100
In addition to recognizing the external and internal states based on each sensor data stored in M11, the behavior of the pet robot 1 is determined based on the recognition result and a notification given from the center-of-gravity calculation image processing object 86 (FIG. 8). It is the module that decides, "joy", "sadness", "surprise",
For a total of six emotions, “fear”, “disgust”, and “anger”, each emotion has a parameter indicating the strength of the emotion.

Then, the action determination module 100 recognizes the values of the parameters of each of these emotions as the recognized external and internal states and the image processing object 86 for calculating the center of gravity.
, And the next action is determined based on the values of the parameters of these emotions and the recognized external and internal states, and an action command based on the determination result. To the motion generation module 101 or the tracking module 1 of the middleware layer 70
02.

The motion generating module 101 and the tracking module 102 generate a servo command value to be given to each of the actuators 25 corresponding to the action based on the given action command, and use these as drive signals described above. The virtual robot 63 of the robotic server object 62 and the signal processing circuit 14 sequentially transmit the signals to the corresponding actuators 25.

As a result, for example, when a motion is suddenly detected, the parameter of “surprise” in the action generation module 71 is increased by the stimulus, so that the action generation module 71 sets “surprise” as the next action. Is determined. Then, the action command corresponding to the action determined in this way is sent to the action generation module 70.
The operation generation module 70 generates a servo command value or the like based on the action command, and the servo command value or the like is provided to the corresponding actuator 25 or the like as the above-described drive signal, whereby an action that shows surprise is expressed.

If the action generation module 100 determines an action that tracks the motion of the moving object,
The command according to the determined action is sent to the tracking module 10.
2, the tracking module 102 generates a servo command value or the like based on the action command, and the generated servo command value is supplied to the corresponding actuator 25 or the like as the above-described drive signal, whereby the pet robot 1 shows interest in the moving object. The behavior of turning the head over there or walking and chasing is manifested.

As described above, in the pet robot 1, the detection result of the movement can be reflected on its own behavior.

In the case of the pet robot 1, once the tracking module 102 receives the action command from the action determining module 100, the tracking module 102 transmits necessary information to the image processing object 86 for calculating the center of gravity without passing through the action determining module 100. , So that high-speed tasks such as tracking can be reliably performed.

(6) Operation and Effect of the Present Embodiment In the above configuration, the pet robot 1 detects the moving amount of the entire image between frames and performs a matching process in consideration of the moving amount to determine the moving object in the image. Is detected.

Accordingly, the pet robot 1 can accurately detect a moving object while efficiently preventing erroneous detection of the moving object due to the operation of the pet robot 1.

In this pet robot 1, one C
Since the moving object can be detected only by the CD camera 20, the configuration of the pet robot 1 as a whole can be simplified.

According to the above configuration, the movement amount of the entire image between the frames is detected, and the moving object in the image is detected by the matching processing in consideration of the movement amount. The moving object can be detected with high accuracy while preventing the erroneous detection of the moving object beforehand and efficiently. In this way, it is possible to realize a pet robot that can prevent unnatural behavior due to erroneous detection of a moving object and improve entertainment characteristics.

(7) Other Embodiments In the above-described embodiment, a case has been described in which the present invention is applied to the four-legged walking type pet robot 1. However, the present invention is not limited to this. The present invention can be widely applied to various other image processing devices mounted on other devices, pet robots other than the quadruped walking type, and robots other than the pet robot.

In the above embodiment, the CCD
The first detection means for detecting the movement amount of the entire image based on the image information (image data) output from the camera 20 is a motion detection image processing object 83 having a software configuration.
And a hardware configuration of the IPE 35, the present invention is not limited to this, and all of them may be configured as a software configuration or a hardware configuration. Various other configurations can be widely applied.

Further, in the above-described embodiment, the second detection means for detecting a motion in an image by a predetermined motion detection process in consideration of the detection result of the first detection means is a software-structured motion detection. Image processing object 83 and image processing object 86 for calculating the center of gravity,
Although a case has been described in which the configuration is made with the IPE 35 having a hardware configuration, the present invention is not limited to this.
All of these may be configured as a software configuration or a hardware configuration, and various other configurations can be widely applied as the configuration of the second detection unit.

Further, in the above-described embodiment, as a method of detecting the movement amount of the entire image between the previous frame and the current frame, the previous frame image 92 and the current frame image 9 are used.
3 is divided into four parts, and the divided areas of the previous frame image 92 (first to fourth template divided areas 92A _{1 to} 92A _{1 to} 92
And A _4), it has been dealt with the case where the matching process and the corresponding divided region of the current frame image 93 (first to fourth search image segmentation region 93A ₁ ~93A ₄₎ of the present invention is to However, the matching process may be performed without dividing the previous frame image 92 and the current frame image 93 or by dividing the previous frame image 92 and the current frame image 93 into numbers other than four.

[0133]

As described above, according to the present invention, in the image processing apparatus and method, the moving amount of the entire image based on the image information is detected, and the predetermined motion detecting process in consideration of the detection result is performed. By detecting the movement of the image, it is possible to realize an image processing apparatus and method capable of accurately detecting a moving object in an image regardless of the presence or absence of movement of the entire image.

Further, according to the present invention, in the robot apparatus and the control method thereof, the moving amount of the entire image based on the image information output from the image pickup means for picking up the outside is detected, and the predetermined amount is determined in consideration of the detection result. By detecting the motion in the image by the motion detection process and reflecting the detection result in the action, the image output from the imaging unit regardless of the presence or absence of the movement of the imaging unit due to the operation of the robot apparatus A robot device capable of accurately detecting a moving object in an image based on information and performing an action reflecting the detection result, and thus improving entertainment characteristics, and a control method thereof can be realized.

[Brief description of the drawings]

FIG. 1 is a perspective view showing a configuration of a pet robot according to the present embodiment.

FIG. 2 is a block diagram illustrating a circuit configuration of the pet robot.

FIG. 3 is a block diagram illustrating a configuration of a signal processing circuit.

FIG. 4 is a block diagram for explaining a method of transferring image data in the pet robot.

FIG. 5 is a conceptual diagram showing a software configuration of a control program.

FIG. 6 is a flowchart illustrating an image processing procedure.

FIG. 7 is a flowchart illustrating an image processing procedure.

FIG. 8 is a block diagram for explaining an image processing method in the pet robot.

FIG. 9 is a conceptual diagram serving to provide an overview of motion detection in a pet robot.

FIG. 10 is a flowchart illustrating a moving object detection processing procedure.

FIG. 11 is a conceptual diagram serving to provide an overview of a moving object detection procedure in a pet robot.

FIG. 12 is a conceptual diagram explaining how to detect the movement amount of the entire image in the pet robot.

FIG. 13 is a block diagram for explaining the operation of reflecting a moving object detection result in an action.

[Explanation of symbols]

1 ... pet robot, 10 ... CPU, 11 ... DR
AM, 14 ... signal processing circuit, 20 ... CCD camera,
34 FBK / CDT, 35 IPE, 61A
FBK / CDT driver, 63: virtual robot, 83: motion detection image processing object, 86:
Centroid calculation image processing object 92 ...... previous frame image, 92A ...... central area, 92A ₁ ~92A ₄ ...... template divided regions, 93 ...... current frame image, 93A ₁
... 93A ₄ ... Search image divided area, 93B... Image area, 100... Action determination module, 101... Action generation module, 102... Tracking module, RT2.
Moving object detection processing procedure.

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06T 1/00 300 G06T 1/00 300 7/00 300 7/00 300D F-term (Reference) 2C150 CA01 CA02 CA04 DA04 DA05 DA24 DA25 DA26 DA27 DA28 DF03 DF04 DF33 ED42 ED47 ED52 EF03 EF07 EF13 EF16 EF23 EF29 EF34 EF36 3C007 AS36 CS08 KS18 KS23 KS24 KS24 KS27 KS31 KS39 KT01 KT11 WA04 DC05 BA16 DC06 A05 5A02 DC FA60 GA08 GA19 HA04 JA03 JA09

Claims (12)

[Claims] A first detecting means for detecting a movement amount of the entire image based on the image information; and a predetermined motion detecting process considering a detection result of the first detecting means, to detect a motion in the image. An image processing apparatus, comprising: 2. The method according to claim 1, wherein the first detecting means calculates a matching score between a divided image of a current frame based on the image information and a divided image at a corresponding position of a previous frame based on the image information while shifting the matching score in pixel units. And
The image processing apparatus according to claim 1, wherein a movement amount of the entire image is detected based on a calculation result. 3. The image processing apparatus according to claim 1, wherein said second detecting means changes the detection sensitivity of the movement in the image according to the detected movement amount. 4. A first step of detecting a movement amount of the entire image based on the image information, and a second step of detecting a movement in the image by performing a predetermined movement detection process in consideration of the detected movement amount. An image processing method comprising: 5. In the first step, a matching score between a divided image of a current frame based on the image information and a divided image at a corresponding position of a previous frame based on the image information is calculated while being shifted in pixel units. ,
The image processing method according to claim 4, wherein a movement amount of the entire image is detected based on a calculation result. 6. The image processing method according to claim 4, wherein, in the second step, a detection sensitivity of the movement in the image is changed according to the detected movement amount. 7. A robot apparatus for autonomously determining and expressing an action, comprising: an image pickup means for picking up an image of the outside; and a first detection for detecting a movement amount of the entire image based on image information output from the image pickup means. Means, and second detection means for detecting a motion in the image by a predetermined motion detection process in consideration of the detection result of the first detection means, wherein the detection result of the second detection means is A robot device characterized in that it is reflected in an action. 8. The first detecting means calculates a matching score between a divided image of a current frame based on the image information and a divided image at a corresponding position of a previous frame based on the image information while shifting the matching score in pixel units. And
The robot apparatus according to claim 7, wherein a movement amount of the entire image is detected based on a calculation result. 9. The robot apparatus according to claim 7, wherein said second detecting means changes the detection sensitivity of the movement in the image according to the detected movement amount. 10. A control method for a robot apparatus which autonomously determines and expresses an action, wherein an image of the outside is taken by an image pickup means mounted on the robot, and an entire image is formed based on image information output from the image pickup means. A first step of detecting a movement amount of the image, a second step of detecting a movement in the image by a predetermined motion detection process in consideration of the detected movement amount, and a detection result of the second step Reflecting in the behavior of the robot device a third step. 11. In the first step, a matching score between a divided image of a current frame based on the image information and a divided image at a position corresponding to a previous frame based on the image information is calculated while being shifted in pixel units. ,
The method according to claim 10, wherein a movement amount of the entire image is detected based on a calculation result. 12. The method according to claim 10, wherein in the second step, the detection sensitivity of the movement in the image is changed according to the detected movement amount.