JP2021174089A - Information processing device, information processing system, information processing method and program - Google Patents

Abstract

To provide an information processing device, an information processing system, an information processing method and a program which efficiently determine the occurrence of concealment of an object in an image.SOLUTION: In a three-dimensional measurement system 100, an information processing device 110 includes: an image input unit 210 which acquires a plurality of images in which an object is photographed from different viewpoints; a centroid calculation unit 232 which calculates a centroid position of the object in the image, based on a partial area of the acquired image; and a concealment determination unit 240 which determines whether concealment of the object has occurred in each image, based on a characteristic value relevant to a time change of the centroid position.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to information processing devices, information processing systems, information processing methods and programs.

Conventionally, three-dimensional measurement of an animal body has been performed for assembling parts in a factory and for highly accurate alignment adjustment. For three-dimensional measurement, those using an imaging system including a stereo camera are known.

In the above-mentioned three-dimensional measurement, an image of an object taken by a stereo camera is used. In 3D measurement, when the environment or the object fluctuates from moment to moment, occlusion (an event in which the object is hidden by an obstacle in the image) often occurs, and it may be difficult to calculate the distance of the object. There is sex.

In relation to occlusion, for example, the prior art disclosed in Japanese Patent Application Laid-Open No. 2003-150940 (Patent Document 1) is known. Patent Document 1 discloses a stereo image processing apparatus that searches for corresponding points using a reference image from one reference camera and a comparison image from at least one comparison camera. Patent Document 1 detects a camera pair in which occlusion occurs based on the degree of difference between the target pixel in the reference image and the corresponding pixel value in the comparison image, and excludes the camera pair from the corresponding point search. The configuration to be used is disclosed. However, Patent Document 1 does not disclose in terms of efficiently determining the occurrence of concealment of an object in an image.

The present disclosure has been made in view of the above points, and an object of the present disclosure is to provide an information processing apparatus capable of efficiently determining the occurrence of concealment of an object in an image.

In the present disclosure, in order to solve the above problems, an information processing device having the following features is provided. The information processing device includes acquisition means for acquiring a plurality of images of an object taken from different viewpoints. The information processing device also includes a calculation means for calculating the position of the center of gravity of the object in the image based on a partial region of the image. The information processing apparatus includes a determination means for determining whether or not concealment of an object has occurred in each image based on a characteristic value related to a change in the position of the center of gravity with time.

With the above configuration, it is possible to efficiently determine the occurrence of concealment of an object in an image.

Overall configuration diagram of a three-dimensional measurement system according to one or more embodiments. The figure explaining the occlusion (concealment by an obstacle of an object) which may occur in a 3D measurement system. A hardware configuration diagram of a personal computer that can be used as a three-dimensional measuring device according to one or more embodiments. The functional block diagram of the 3D measuring apparatus according to 1st Embodiment. The flowchart which shows the 3D measurement process which performs 3D measurement while dealing with occlusion, which is executed by 3D measurement apparatus by 1st Embodiment. The figure explaining the initial setting of the region of interest and the exclusion of an image from a stereo pair in a 3D measuring apparatus according to a specific embodiment. The schematic diagram explaining the tracking method of the object in the 3D measuring apparatus by a specific embodiment. The graph which schematically represents the time series of the center of gravity position and the center of gravity velocity generated by the three-dimensional measuring apparatus according to the first embodiment. The figure which shows the state of tracking when the brightness of an object and an obstacle is the same. The figure which shows the state of tracking when the brightness of an obstacle is lower than the brightness of an object. A graph showing the position of the center of gravity and the speed of the center of gravity actually measured when the object is stationary and the obstacle is moving. The flowchart which shows the 3D measurement process which performs 3D measurement while dealing with occlusion, which is executed by 3D measurement apparatus by 2nd Embodiment. The graph which schematically represents the time series of the center of gravity position and the center of gravity velocity generated by the three-dimensional measuring apparatus according to the second embodiment. A graph showing the acceleration of the center of gravity actually measured when the object is stationary and the obstacle is moving. The figure explaining the occlusion (concealment by an obstacle of an object) which can occur by two or more high-speed vision cameras in a three-dimensional measurement system. The flowchart which shows the 3D measurement process which performs 3D measurement while dealing with occlusion, which is executed by 3D measurement apparatus by 3rd Embodiment. The flowchart which shows a part of 3D measurement processing which performs 3D measurement while dealing with occlusion, which is executed by 3D measurement apparatus by 4th Embodiment.

Hereinafter, embodiments of the present invention will be described, but the embodiments of the present invention are not limited to the embodiments described below. In the embodiment described below, as an example of the information processing system and the information processing device, each includes a plurality of high-speed vision cameras 160 and a three-dimensional measuring device 110 that performs three-dimensional measurement of an object. This will be described with reference to the three-dimensional measurement system 100 and the three-dimensional measurement device 110.

FIG. 1 shows the overall configuration of the three-dimensional measurement system 100 according to one or more embodiments. As shown in FIG. 1, the three-dimensional measurement system 100 includes a three-dimensional measurement device 110, a camera synchronization device 150, and a plurality of high-speed vision cameras 160a to 160z connected to the three-dimensional measurement device 110 and the camera synchronization device 150, respectively. It is composed including and.

The plurality of high-speed vision cameras 160a to 160z are arranged at different positions so as to face a predetermined direction, and are configured to shoot the object 102 as a subject from different viewpoints. Therefore, parallax occurs between a plurality of images captured by the plurality of high-speed vision cameras 160a to 160z. Preferably, each of the plurality of high-speed vision cameras 160a to 160z has a field of view set so as to cover a movable range of the object 102. Each high-speed vision camera 160 photographs an object 102 in the field of view based on a timing signal input from the camera synchronization device 150, and inputs the captured image to the three-dimensional measuring device 110.

The high-speed vision camera 160 is an imaging device that captures an image at a higher speed than a normal camera of about 30 fps to 60 fps, and typically, one that realizes a frame rate of about 1000 fps is preferably used. Due to this high frame rate, the high-speed vision camera 160 can handle even high-speed moving animals. Since the shutters of each high-speed vision camera 160 are synchronized by the external camera synchronization device 150, it is possible to shoot the same frame at the same time.

The camera synchronization device 150 externally controls the shutter synchronization of each high-speed vision camera 160. By connecting the external camera synchronization device 150 to a plurality of high-speed vision cameras 160 and setting the master-slave in the program, shutter synchronization can be performed by the input / output of the external trigger.

The three-dimensional measuring device 110 performs three-dimensional measurement including the depth of the object 102 based on a plurality of images obtained by photographing the object 102 with a plurality of high-speed vision cameras 160a to 160z. As will be described in detail later, the three-dimensional measuring device 110 according to the present embodiment does not conceal the object 102 by an obstacle (hereinafter, may be referred to as occlusion) when performing the three-dimensional measurement. Is determined, and the image of the high-speed vision camera 160 in which occlusion has occurred is excluded from the image group used in the three-dimensional measurement.

FIG. 2 illustrates occlusions that can occur in a three-dimensional measurement system. FIG. 2A shows a state in which the object 102 is photographed by the first to third high-speed vision cameras 160a to 160c at a certain time point. FIG. 2B shows a state in which the object 102 is photographed by the first to third high-speed vision cameras 160a to 160c in a state where the object 102 is moving after the time point of FIG. 2A. ..

At the time of FIG. 2A, in the images 170a to 170c taken by each of the three high-speed vision cameras 160a to 160c, the obstacle 104 is the second high-speed vision as depicted in FIG. Although within the field of view of the camera 160b, occlusion has not occurred in any of the high speed vision cameras 160.

However, when the object 102 moves and the state shown in FIG. 2B is reached, the object 172b is hidden by the obstacle 174b in the image 170b of the second high-speed vision camera 160b, and the occlusion occurs. It has occurred. At this time, because the object 102 is hidden by the obstacle 104, the image 170b captured by the second high-speed vision camera 160b is excluded from the image group used in the three-dimensional measurement (in the stereo method). , Do not use as a stereo pair). The three-dimensional measuring device 110 shown in FIG. 1 is configured to efficiently determine the occurrence of occlusion of the object 102 due to some cause and exclude the image of the high-speed vision camera 160 in which the occlusion has occurred.

In the embodiment described below, the obstacle 104 is sufficiently small with respect to the field of view of the high-speed vision camera 160, and at least two of the plurality of high-speed vision cameras 160 are arranged so as not to cause occlusion. Continue the explanation as if it were. Therefore, in the example using the three high-speed vision cameras 160a to 160c of FIG. 2, only one high-speed vision camera 160 causes occlusion.

Further, for convenience of explanation, the object 102 will be described by taking the case where it is a marker made of a retroreflective material as an example, but the marker is attached to another object (for example, a part in assembling parts in a factory). It may be attached, and the attached marker may be detected. Further, the recognized object 102 is not limited to the marker created by the retroreflective material, and any form of the object can be the object as long as it can be recognized from the captured image. Can be done.

Hereinafter, the hardware configuration of the three-dimensional measuring device 110 will be described before the details of the functions of the three-dimensional measuring system 100 are described.

FIG. 3 shows a hardware configuration of a personal computer that can be used as the three-dimensional measuring device 110 according to the present embodiment. Here, the hardware configuration of the personal computer 500 will be described.

As shown in FIG. 3, the personal computer 500 is constructed by a general-purpose computer, and as shown in FIG. 3, the CPU (Central Processing Unit) 501, the ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, HDD (Hard Disk Drive) 504, HDD controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, data bus 510, keyboard 511, pointing device 512, It is equipped with a DVD-RW (Digital Versatile Disk Rewritable) drive 514 and a media I / F 516.

Of these, the CPU 501 controls the operation of the entire personal computer 500. The ROM 502 stores a program used for driving the CPU 501 such as an IPL (Initial Program Loader). The RAM 503 is used as a work area of the CPU 501. HDD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data to the HDD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a printer, or the like. The network I / F 509 is an interface for performing data communication using a communication network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG.

Further, the keyboard 511 is a kind of input means including a plurality of keys for inputting characters, numerical values, various instructions and the like. The pointing device 512 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 514 controls reading or writing of various data to the DVD-RW media 513 as an example of the removable recording medium. In addition, it is not limited to DVD-RW, and may be DVD-R or the like. The media I / F 516 controls reading or writing (storage) of data to a recording medium 515 such as a flash memory.

Hereinafter, the functional configuration of the three-dimensional measuring device 110 will be described with reference to FIG. FIG. 4 shows a functional block of the three-dimensional measuring device 110 according to the first embodiment.

As shown in FIG. 4, the three-dimensional measuring device 110 is input to an image input unit 210 that receives images input from different viewpoints from a plurality of high-speed vision cameras (indicated as HSV in FIG. 4) 160. The target search unit 220 that searches for a predetermined object 102 in the image, the tracking unit 230 that tracks the object 102 in the image, and the occurrence of occlusion of the object 102 due to the obstacle 104 in the image are determined. A concealment determination unit 240, a three-dimensional measurement unit 250 that performs three-dimensional measurement based on a plurality of images from different viewpoints input to the image input unit 210, and a calibration unit 260 that performs calibration for three-dimensional measurement. It is composed including and.

The image input unit 210 receives the input of the images captured by each of the plurality of high-speed vision cameras 160, and sequentially passes the acquired images of the plurality of high-speed vision cameras 160 to the target search unit 220. Time-series data composed of a plurality of frames of images captured by each of the high-speed vision cameras 160 will be input for the number of high-speed vision cameras 160.

The target search unit 220 searches for a predetermined target object 102 from each image. When the object 102 can be identified from the background or another object as a high-luminance region, it can be binarized, for example, and the outline of the object can be searched in the binarized image.

The tracking unit 230 tracks the position of the object 102 in the image. More specifically, the tracking unit 230 includes a center of gravity calculation unit 232 and an area setting unit 234.

The center of gravity calculation unit 232 calculates the position of the center of gravity of the object 102 in each frame based on the image information of a predetermined partial region referred to as a region of interest (ROI). The region of interest can be set as a rectangular region having a size corresponding to the contour of the object 102 obtained by the target search unit 220. By performing image processing only within the region of interest, the amount of calculation can be significantly reduced. The position of the center of gravity is measured in the coordinate system of the captured image of the high-speed vision camera 160. The position of the center of gravity calculated for each high-speed vision camera 160 is input to the concealment determination unit 240 as time-series data. In the present embodiment, the center of gravity calculation unit 232 constitutes a calculation means for calculating the position of the center of gravity of the object.

The area setting unit 234 sets the region of interest (ROI) centered on the position of the center of gravity obtained by the center of gravity calculation unit 232. The region of interest is set with respect to the current frame based on the position of the center of gravity calculated in the frame at the past time point (typically the previous time), and is updated from time to time. In the present embodiment, the area setting unit 234 constitutes a setting means for setting an area of interest in the image based on the position of the center of gravity of the object.

The concealment determination unit 240 determines whether or not occlusion has occurred in each of the plurality of images captured by the plurality of high-speed vision cameras 160, and excludes the images in which the occlusion has occurred. More specifically, the concealment determination unit 240 according to the first embodiment to be described includes a center of gravity velocity calculation unit 242 and a stereo pair selection unit 244.

The center-of-gravity velocity calculation unit 242 calculates the center-of-gravity velocity of each high-speed vision camera 160 based on the time-series data of the given center-of-gravity position. The center of gravity velocity is also calculated as time series data. The concealment determination unit 240 determines whether or not occlusion has occurred based on the time change of the center of gravity velocity calculated by the center of gravity velocity calculation unit 242. The concealment determination unit 240 constitutes a determination unit for determining whether or not occlusion has occurred. In the first embodiment to be described, the change in the velocity of the center of gravity of the object is used in the occlusion determination as a characteristic value related to the time change in the position of the center of gravity in the present embodiment. The determination of the presence or absence of occlusion can be based on the threshold condition for the change in the velocity of the center of gravity.

The stereo pair selection unit 244 excludes the image determined to have caused occlusion from the plurality of images obtained by the plurality of high-speed vision cameras 160, and from the remaining image group, the stereo pair used in the three-dimensional measurement. Select. The stereo pair selection unit 244 constitutes, in the present embodiment, a selection means for selecting at least two images to be used in the three-dimensional measurement.

The three-dimensional measurement unit 250 performs three-dimensional measurement using the principle of triangulation. For the three-dimensional measurement, the stereo method can be used in the embodiments described. The stereoscopic three-dimensional measurement is performed based on the parallax of the feature points obtained from the two images obtained by the high-speed vision camera 160. In the case of the stereo method, only a part having characteristic points needs to be compared, and the calculation method is simple, so that the processing time can be shortened. The three-dimensional measurement unit 250 constitutes a measurement means that performs three-dimensional measurement based on at least two images taken from different viewpoints at substantially simultaneous points selected.

The camera synchronization device 150 synchronizes the shutter of each high-speed vision camera 160 from the outside.

The calibration unit 260 performs a calibration process for each high-speed vision camera 160. More specifically, the calibration unit 260 includes a correction conversion matrix calculation unit 262 and a basic matrix calculation unit 264. The correction conversion matrix calculation unit 262 calculates the correction conversion matrix. The basic matrix calculation unit 264 calculates the basic matrix. When the correction conversion matrix and the basic matrix are obtained, the projection matrix Pn required for the three-dimensional measurement is obtained.

In the stereoscopic three-dimensional measurement, the distance at the pixel position is derived by solving the simultaneous equations of the following equations (1) and (2). In the following equation, m ^~ is the homologous form of the image coordinates (u, v) (center of gravity position), P is the projection matrix, and X ^~ _w are the world coordinates (X _w , Y _w , Z _w). ) Is an isomorphic form. In addition, variables with tilde (~) accents in the following equations (1) and (2) are written as ^{m ~} or X ^~ _{w in the text.} Further, since the three-dimensional measurement by the stereo method and the calibration process for that purpose can be appropriately performed by using the existing technique, no further detailed explanation will be given.

Hereinafter, with reference to FIG. 5, a process of performing three-dimensional measurement while dealing with occlusion, which is executed by the three-dimensional measuring device 110 according to the first embodiment, will be described. FIG. 5 is a flowchart showing a three-dimensional measurement process.

The three-dimensional measurement process shown in FIG. 5 is a process for determining occlusion by utilizing the fact that the position of the center of gravity of an object suddenly changes when an animal body is an object and an obstacle overlaps the object and occlusion occurs. be. In FIG. 5, n high-speed vision cameras 160 will be used.

The process shown in FIG. 5 starts from step S100 in response to, for example, an instruction from the operator to start three-dimensional measurement.

In step S101, the three-dimensional measuring device 110 performs the initial setting of each high-speed vision camera 160 by the calibration unit 260, obtains the correction conversion matrix and the basic matrix, and obtains the projection matrix Pn required for the three-dimensional measurement.

In step S102, the three-dimensional measuring device 110 acquires a plurality of images of the first frame from the plurality of high-speed vision cameras 160. In step S103, the three-dimensional measuring device 110 searches the entire image for the contour of the object and derives the area of each high-speed vision camera 160. In a particular embodiment, in each image, the entire image is binarized and the contour of the object is searched. Here, since a plurality of areas including noise are derived, the object having the maximum area is set as the object shown in the image. In step S104, the three-dimensional measuring device 110 compares the maximum area obtained in step S103 between the images of the high-speed vision cameras 160, and excludes those that are significantly different. In step S105, the three-dimensional measuring device 110 sets an area of interest (ROI) of a size determined according to the area of each of the remaining images.

In the embodiment to be described, those having a significantly different maximum area between the images of the high-speed vision cameras 160 have been excluded. However, it is not limited to this. In another embodiment, when the object is a sphere or the like, an area of interest may be set for all images, and an area of interest that does not correspond to a substantially square may be excluded.

FIG. 6 illustrates the initialization of the region of interest and the exclusion of images from stereo pairs in a 3D measuring device according to a particular embodiment. In FIG. 6, in each image, the contour region of the object 172 is obtained, and the central image having a significantly different maximum area is excluded. Further, the region of interest (ROI) 176 is set so as to include each contour region.

Referencing FIG. 5 again, in step S106, the three-dimensional measuring device 110 selects a stereo pair from the images of the remaining N-1 high-speed vision cameras 160. The selection of the stereo pair in step S105 can be made based on the angle at which the two selected high-speed vision cameras 160 are installed. It is assumed that the arrangement and the direction of the field of view of each high-speed vision camera 160 are appropriately set in advance and given to the three-dimensional measuring device 110. The wider the distance (baseline length) between the stereo pair cameras, the larger the parallax, so the measurement accuracy of the depth (Z direction) can be improved, but on the contrary, the resolution in the image plane (XY direction) decreases. It will be. Therefore, it is preferable to select a pair having an angle close to about 90 degrees.

In step S107, the three-dimensional measuring device 110 calculates the position of the center of gravity (x, y) of the object based on the image information of the region of interest in each of the images of the stereo pair, and also in the X direction and the Y direction of the center of gravity position. The center of gravity velocity v is calculated from the amount of change in.

In step S108, the three-dimensional measuring device 110 derives the change v1 of the center-of-gravity velocity based on the center-of-gravity velocity v obtained in step S107, and compares it with the predetermined threshold value α1. If the change v1 of the velocity of the center of gravity is equal to or greater than the predetermined threshold value α1, the position of the center of gravity (x, y) is suddenly changed, and it is determined that occlusion has occurred.

If it is determined in step S108 that the change v1 of the center of gravity velocity is equal to or greater than the predetermined threshold value α1, the process is branched to step S112. In step S112, the three-dimensional measuring device 110 excludes the image determined to have occlusion, acquires an image from the remaining N-1 high-speed vision cameras 160, and then returns the process to step S106. In the embodiment shown in FIG. 5, when it is determined that there is occlusion, the process proceeds to the next frame.

On the other hand, as a result of comparing the change v1 of the center of gravity velocity and the threshold value α1 in step S108, if it is determined that the change v1 of the center of gravity velocity is smaller than the threshold value α1 (YES), it is determined that no occlusion has occurred, and step S109 Processing proceeds to. The details of the threshold value α1 will be described later.

In step S109, the three-dimensional measuring device 110 uses the center of gravity position (x, y) obtained from each of the stereo pair images and the projection matrix Pn obtained in step S101 to perform three-dimensional measurement based on the principle of triangulation. Run. In step S110, the three-dimensional measuring device 110 resets the region of interest centered on the calculated position of the center of gravity of the object. The details of resetting the region of interest will be described later with reference to FIG. 7.

In step S111, the three-dimensional measuring device 110 acquires a plurality of images of the next frame from the two high-speed vision cameras 160 selected as the stereo pair, and loops the process to step S107.

In the above flow, two types of occlusion determination methods are substantially performed in a series of processes in steps S103 to S106 and a series of processes in steps S108, S112 and S106. The former judgment is just a step for initial setting, and the latter is the main occlusion judgment method. While the occlusion is not occurring, the 3D measurement and tracking loop can be rotated using only the position of the center of gravity of each image, so that the calculation is simple and high speed can be realized.

Hereinafter, a method of resetting the region of interest while tracking the object will be described with reference to FIG. 7. FIG. 7 shows an outline for explaining an object tracking method in a three-dimensional measuring device according to a specific embodiment.

As an animal tracking method, a method called Self-Windowing (I. Ishii, et al., "Self Windowing for High Speed Vision", IEEE International Conference on Robotics & Automation, 1999) can be preferably used. , It will be described assuming that tracking is performed by this method.

In FIG. 7, 310, 320, 330, 340, 350 represent images of frames taken in time series with a certain high-speed vision camera 160. The frames 310, 320, 330, 340, and 350 are spaced by a predetermined number of frames instead of one frame.

First, in the first frame 310, the object 312 is photographed as a black object. In the second frame 320, the moment of the object 322 is derived, and the position of the center of gravity (x, y) (represented by the white point 324 in the figure) is calculated. In the third frame 330, the region of interest 336 centered on the position of the center of gravity (x, y) 334 is set. The size of the region of interest 336 is a size that matches the size of the object 332. In the fourth frame 340, the state in which the position of the object 342 is moving over time is shown. The center of gravity position (x, y) 344 is calculated within the region of interest 346 obtained in the third frame 330. In the fifth frame 350, a new region of interest 356 centered on the center of gravity position (x, y) 354 obtained in the fourth frame is set.

Conventionally, it has been the mainstream to calculate and track the entire image. However, in the present embodiment, since the high-speed vision camera 160 is preferably used, the continuity of the image information is high, and it is possible to narrow down the area of interest without performing prediction processing or the like. Therefore, it is not necessary to read the information of the entire image, the processing time can be significantly reduced, and it can be said that the method does not depend on the size of the image. Further, since the setting of the region of interest can be updated only from the image of the region of interest and the input image of the previous frame, simplified correspondence can be realized at the same time.

FIG. 8 is a graph schematically showing a time series of a center of gravity position and a center of gravity velocity generated by the three-dimensional measuring device according to the first embodiment. FIG. 8 illustrates a case where the speed change of the animal body is relatively small. FIG. 8 (A) shows the time course of the center of gravity position (x, y) (absolute position in the screen) with respect to time t, and FIG. 8 (B) shows the center of gravity velocity obtained by synthesizing the change in the center of gravity position. It shows the change of v with time.

As shown in FIG. 8, when the velocity change of the animal body is relatively small, the center of gravity position (x, y) is almost linear until the timing when occlusion occurs, that is, until the time t1 when the obstacle enters the region of interest. It has become. On the other hand, when occlusion occurs, the position of the center of gravity (x, y) fluctuates greatly. Looking at the center-of-gravity velocity v, a minute velocity change v1 occurs until the time t1. However, after the timing at which the occlusion occurs, a change larger than the minute velocity change v1 before the occurrence of the occlusion occurs. Therefore, while monitoring the average center-of-gravity velocity v over a predetermined period, the threshold value ± α1 (> the predetermined upper limit and lower limit larger than the minute change width of the center-of-gravity velocity observed at normal times is used as a reference. ) Is provided, so that the occurrence of occlusion can be suitably captured. The specific value of the threshold value α1 may be selected according to whether the determination sensitivity is prioritized or the prevention of erroneous determination is prioritized.

FIG. 9 shows a state of tracking when the brightness of the object and the obstacle are about the same. When the object and the obstacle have the same brightness as shown in FIG. 9, when the occlusion occurs, the object and the obstacle overlap each other as shown in 392 and 398 and 402 and 408. Is recognized in image processing. Therefore, before the occurrence of occlusion, the center of gravity of the object was detected, and the center of gravity moved at a predetermined speed according to the moving speed of the object. It moves to the position of the center of gravity of an integral object that overlaps with an obstacle, and the position of the center of gravity fluctuates greatly.

FIG. 10 shows a state of tracking when the brightness of an obstacle is lower than the brightness of an object. When the object is significantly different from the brightness of the obstacle as shown in FIG. 10, when occlusion occurs, the object simply hides the object in the obstacle and the area of the object becomes large as shown in 442 and 452. It is recognized as having decreased. Therefore, before the occurrence of occlusion, the center of gravity of the entire object was detected, and the center of gravity moved at a predetermined speed according to the moving speed of the object, whereas immediately after the occurrence of occlusion, an obstacle occurred. It becomes the position of the center of gravity of the part of the object excluding the part hidden by the object, and the position of the center of gravity fluctuates greatly.

Further, as shown in FIGS. 9 and 10, the position of the region of interest moves from the moment when the obstacles 398 and 448 enter the region of interest 396 and 446.

FIG. 11 is a graph showing the measured center of gravity position (x, y) and center of gravity velocity v when the object is stationary and the obstacle is moving. Note that FIG. 11 is based on an experiment in which a recurrence reflector having a diameter of about 10 mm was photographed using a high-speed vision camera having a frame rate of 1000 fps. FIG. 11A shows a time series of the center of gravity position (x, y) and the center of gravity velocity v, and FIG. 11B is an enlarged view of the vertical axis of the center of gravity velocity v. The image was taken while the object was stationary and the obstacle was moving.

As is clear from FIG. 11, until the occlusion occurrence time, the change in the center of gravity position (x, y) and the center of gravity velocity v (shown as the velocity change v1) is minute, while after the occlusion occurs. It can be seen that a large fluctuation has occurred. From this, it is understood that occlusion can be suitably detected by setting a predetermined threshold condition for the change in the center of gravity velocity v with the change in the center of gravity velocity v as the characteristic value.

In the first embodiment described above, the change in the velocity of the center of gravity of the object is used for the occlusion determination as a characteristic value related to the change in the position of the center of gravity with time. However, the characteristic value that can be used for the occlusion determination is not limited to the change in the velocity of the center of gravity of the object. Hereinafter, the three-dimensional measurement process for performing the three-dimensional measurement while coping with the occlusion according to the second embodiment will be described with reference to FIGS. 4 and 12 to 14. In the second embodiment, the change in the acceleration of the center of gravity of the object is used as a characteristic value used for the occlusion determination. Hereinafter, the differences will be mainly described, and unless otherwise specified, the same configuration as that of the first embodiment will be provided.

In the second embodiment, the center-of-gravity velocity calculation unit 242 shown in FIG. 4 calculates the center-of-gravity velocity v and calculates the center-of-gravity acceleration a for each high-speed vision camera 160 based on the calculated time-series data of the center-of-gravity velocity. do. The center of gravity acceleration a is also calculated as time series data. The concealment determination unit 240 determines whether or not occlusion has occurred based on the time change a2 of the center of gravity acceleration calculated by the center of gravity velocity calculation unit 242. In the second embodiment to be described, the change a2 of the acceleration of the center of gravity of the object is used in the occlusion determination as a characteristic value related to the time change of the position of the center of gravity in the present embodiment. The determination of the presence or absence of occlusion can be based on the threshold condition for the change a2 of the center of gravity acceleration.

Hereinafter, with reference to FIG. 12, a process of performing three-dimensional measurement while dealing with occlusion, which is executed by the three-dimensional measuring device 110 according to the second embodiment, will be described. FIG. 12 is a flowchart showing a three-dimensional measurement process according to the second embodiment.

The three-dimensional measurement process shown in FIG. 12 utilizes the fact that the position of the center of gravity of an object suddenly changes when an obstacle overlaps the object and occlusion occurs, targeting an animal body in which the change in the velocity of the center of gravity is relatively large. Then, it is a process of performing an occlusion determination. In FIG. 12, n high-speed vision cameras 160 will be used.

The processes of steps S200 to S206 and the processes of steps S209 to S212 are the same as the processes of steps S100 to S106 and the processes of steps S109 to S112 shown in FIG. explain.

In step S207, the three-dimensional measuring device 110 calculates the center of gravity position (x, y) and the center of gravity velocity v of the object based on the image information of the region of interest in each of the images of the stereo pair, and further, the center of gravity acceleration a. To calculate.

In step S208, the three-dimensional measuring device 110 derives the change a2 of the center of gravity acceleration based on the center of gravity acceleration a obtained in step S207 and compares it with the predetermined threshold value α2. If the change a2 of the acceleration of the center of gravity is equal to or greater than the predetermined threshold value α2, the position of the center of gravity (x, y) is suddenly changed, and it is determined that occlusion has occurred. If it is determined in step S208 that the change a2 of the center of gravity acceleration is equal to or greater than the predetermined threshold value α2 (NO), the process is branched to step S212. As a result of comparing the change a2 of the center of gravity acceleration and the threshold value α2 in step S208, if it is determined that the change a2 of the center of gravity acceleration is smaller than the threshold value α2 (YES), it is determined that no occlusion has occurred, and the process proceeds to step S209. Processing proceeds.

FIG. 13 is a graph schematically showing the time series of the center-of-gravity velocity and the center-of-gravity acceleration generated by the three-dimensional measuring device according to the second embodiment. FIG. 13 illustrates a case where the speed change of the animal body is relatively large. FIG. 13 (A) shows the time-dependent change of the center-of-gravity velocity v with respect to time t, and FIG. 13 (B) shows the time-dependent change of the center-of-gravity acceleration a. The gray line indicates the time when occlusion does not occur, and the black line exemplifies the time when occlusion occurs.

When the speed change of the animal body is large, when occlusion occurs, that is, when an obstacle enters the region of interest, the speed of the center of gravity fluctuates as shown in FIG. 13 (A). Since the speed change v3 is small immediately before the time t2, the occlusion can be determined from the speed change v3, but if the occlusion occurs when the speed change v2 is large like just before the time t1, the speed change v2 alone It becomes difficult to set a threshold. As shown in FIG. 13B, the center-of-gravity acceleration a has a minute acceleration change a2 until the time t1 when occlusion does not occur. However, when occlusion occurs, a fluctuation larger than the change a2 of the acceleration of the center of gravity occurs.

Therefore, while monitoring the average center-of-gravity acceleration a over a predetermined period, the threshold value ± α2 (a predetermined upper limit and lower limit larger than the minute change width a2 of the center-of-gravity acceleration observed at normal times is used as a reference. ) Is provided, it is possible to suitably capture the occurrence of occlusion. The specific value of the threshold value α2 may be selected according to whether the determination sensitivity is prioritized or the prevention of erroneous determination is prioritized.

FIG. 14 is a graph showing the measured center-of-gravity acceleration a when the object is stationary and the obstacle is moving. It should be noted that the image was taken in a state where the object was stationary and the obstacle was moving, and the experimental conditions were the same as in the case of FIG.

As is clear from FIG. 14, the change in the center-of-gravity acceleration a (indicated as the acceleration change a2) is small until the occlusion occurrence time, and on the other hand, a large change occurs after the occlusion occurrence. Recognize. From this, in consideration of the movement of the object at a non-constant velocity, the occlusion is suitably detected by setting a predetermined threshold condition for the change of the center of gravity acceleration a with the change of the center of gravity acceleration a as a characteristic value. It is understood that it is possible. Even when the brightness of the obstacle is low, the position of the center of gravity, the velocity of the center of gravity, and the acceleration of the center of gravity of the region of interest show similar fluctuations, so that the determination can be made by the same method.

In the first and second embodiments described above, the calculation is simple because the three-dimensional measurement and tracking loop can be rotated using only the position of the center of gravity of each image while occlusion is not occurring. It was possible to achieve high speed. On the other hand, it was assumed that there was only one high-speed vision camera 160 in which occlusion occurred. Hereinafter, a three-dimensional measurement process for performing three-dimensional measurement while coping with the occlusion according to the third embodiment will be described with reference to FIGS. 15 and 16. The third embodiment can deal with the possibility that a plurality of high-speed vision cameras 160 occur in which occlusion occurs.

Hereinafter, the differences will be mainly described, and unless otherwise specified, the same configuration as that of the second embodiment will be provided. Further, although the third embodiment will be described as being based on the second embodiment, it goes without saying that similar changes may be applied to the first embodiment.

FIG. 15 illustrates occlusion (concealment of an object by an obstacle) that can occur in two or more high-speed vision cameras in a three-dimensional measurement system. In the first embodiment and the second embodiment, it is assumed that there is one camera that causes occlusion, but in the third embodiment, the possibility that a plurality of cameras that cause occlusion occur is also dealt with. ..

For example, as shown in FIG. 15A, the object moves in a state where occlusion occurs in the second high-speed vision camera 160b and occlusion does not occur in the other high-speed vision cameras 160a, 160c, 160d. Suppose you did. Then, as shown in FIG. 15B, as a result of the movement of the object 102, an obstacle is also reflected in the third high-speed vision camera 160c, and the second is determined to have occlusion. Even with the high-speed vision camera 160b, occlusion may still occur. In such a case, since the determination from the state with occlusion to the state without occlusion is not performed, it is necessary to reset the region of interest for the object 102. Here, with the remaining high-speed vision cameras 160a, 160b, 160d excluding the third high-speed vision camera 160c newly determined to have occlusion, the contour of the object is searched again and the region of interest is newly set. ..

Then, the region of interest is compared with each high-speed vision camera 160, and those having clearly different sizes are determined to have occlusion and are excluded from the stereo pair. This makes it possible to select a new stereo pair excluding the high-speed vision camera 160 in which occlusion has occurred. Since the process of searching for the pixels of the entire image takes more time than the measurement of the position of the center of gravity, high speed can be realized by removing the process from the normal loop.

Hereinafter, with reference to FIG. 16, a process of performing three-dimensional measurement while dealing with occlusion, which is executed by the three-dimensional measuring device 110 according to the third embodiment, will be described. FIG. 16 is a flowchart showing a three-dimensional measurement process according to the third embodiment.

The processing content itself of steps S300 to S312 is the same as the processing of steps S200 to S212 shown in FIG. On the other hand, the position to loop when occlusion occurs is different. In the third embodiment, when it is determined in step S308 that the change a2 of the center of gravity acceleration is equal to or greater than the predetermined threshold value α2 (NO), the process is branched to step S312. In step S312, the three-dimensional measuring device 110 excludes the image determined to have occlusion, acquires the image from the remaining N-1 high-speed vision cameras 160, and then returns the process to step S303 instead of step S306. ..

In this way, when it is detected that occlusion has occurred, the object is re-searched with the remaining images excluding the images determined to have occlusion. This makes it possible to select an appropriate stereo pair even when occlusion may occur in a plurality of high-speed vision cameras 160 as shown in FIG. Further, as long as occlusion does not occur, the loop of steps S307 to S311 is performed, so that the process is completed only by the image processing of the region of interest, so that high speed can be realized.

In the first to third embodiments described above, the region of interest was fixed at the initially set size. On the other hand, the object 102 may approach or move away from the high-speed vision camera 160. Hereinafter, a three-dimensional measurement process for performing three-dimensional measurement while coping with the occlusion according to the fourth embodiment will be described with reference to FIG. A fourth embodiment adjusts the size of the region of interest to an appropriate size.

Hereinafter, the differences will be mainly described, and unless otherwise specified, the same configuration as that of the third embodiment will be provided. Further, although the fourth embodiment will be described as being based on the third embodiment, it goes without saying that similar changes may be applied to the first or second embodiment.

FIG. 17 is a flowchart showing a three-dimensional measurement process according to the fourth embodiment. The processes of steps S400 to 410 and the processes of steps SS421 to S413 are the same as the processes of steps S300 to S412 shown in FIG. In the flowchart shown in FIG. 17, the step of step S411 is newly added.

In step S411, the three-dimensional measuring device 110 changes the size of the region of interest according to the change in the distance of the object obtained as a result of the three-dimensional measurement. When the distance is shortened, that is, when the object is close to the high-speed vision camera 160, the size of the region of interest is made larger than the previous time, and when the distance is far, that is, the object is moved away from the high-speed vision camera 160. If so, set the size of the area of interest to be smaller than the previous time. This makes it possible to track the object in the distance direction as well, and even if the object moves three-dimensionally, the size of the region of interest can be minimized. Therefore, the occlusion determination can be performed more accurately. The degree of size change may be determined according to the change in distance.

According to the embodiment described above, it is possible to provide an information processing apparatus, an information processing system, an information processing method and a program capable of efficiently determining the occurrence of concealment of an object in an image.

In particular, according to the three-dimensional measurement according to the above-described embodiment, it is possible to realize high-speed and accurate three-dimensional measurement even when occlusion occurs. Since the occlusion determination is based on the time change of the position of the center of gravity of the object, it is possible to reduce the calculation resources and the processing time as compared with the case where all the image information is compared or all the image information is used. It is possible. In a preferred embodiment, many processes are completed only by image processing within the region of interest, which is even more advantageous in terms of computational resources and processing time. In particular, it is advantageous as compared with a technique that requires comparison calculation of all pixel values in the target range between cameras for occlusion determination.

In the embodiment to be described, as an example of the information processing system, a three-dimensional measurement system 100 including a plurality of high-speed vision cameras 160, a three-dimensional measurement device 110, and a camera synchronization device 150 has been described. However, the information processing system is not limited to the above configuration, and components may be added or deleted. For example, a device having the functions of the three-dimensional measuring device 110 and the camera synchronization device 150 may be provided, and the three-dimensional measuring device 110 and the camera synchronization device 150 may be omitted. In another embodiment, in addition to the plurality of high-speed vision cameras 160, the three-dimensional measuring device 110, and the camera synchronization device 150, a robotic device such as a robot arm that physically operates the object 102, and a control for controlling the robot device. It may be configured to include a controller. At that time, a device having the functions of the three-dimensional measuring device 110 and the control controller may be provided.

In the embodiment described, it is assumed that the object is moving and the obstacle is stationary, but when the object is stationary and the obstacle is moving, or when the object and the obstacle are both moving. Needless to say, it can also be applied when moving. Further, in the embodiment described, it is assumed that n high-speed vision cameras 160 are fixed and the object is moving, but the relative position between the n high-speed vision cameras 160 and the object is described. It is applicable to any configuration as long as the relationship fluctuates.

Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array) and conventional circuit modules.

The group of devices described in the embodiments is only one of a plurality of computing environments for implementing the embodiments disclosed herein. In certain embodiments, the three-dimensional instrument comprises a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including networks, shared memory, and the like, and perform the processes disclosed herein.

Although the embodiments and examples of the present invention have been described above, the embodiments and examples of the present invention are not limited to the above-described embodiments and examples, and other embodiments, other examples, and the like. Additions, changes, deletions, etc. can be made within the range that can be conceived by those skilled in the art, and any aspect is included in the scope of the present invention as long as the actions and effects of the present invention are exhibited.

100 ... 3D measurement system, 102 ... Object, 104 ... Obstacle, 110 ... 3D measurement device, 150 ... Camera synchronization device, 160 ... High-speed vision camera, 170 ... Image, 172 ... Object, 174 ... Obstacle, 210 ... image input unit, 220 ... target search unit, 230 ... tracking unit, 232 ... center of gravity calculation unit, 234 ... area setting unit, 240 ... concealment determination unit, 242 ... center of gravity speed calculation unit, 244 ... stereo pair selection unit, 250 ... 3D measurement unit, 260 ... Calibration unit, 262 ... Correction conversion matrix calculation unit, 264 ... Basic matrix calculation unit, 500 ... Personal computer, 501 ... CPU, 502 ... ROM, 503 ... RAM, 504 ... HDD, 505 ... HDD controller, 506 ... display, 508 ... external device connection I / F, 509 ... network I / F, 511 ... keyboard, 512 ... pointing device, 514 ... DVD-RW drive, 513 ... DVD-RW media, 516 ... media I / F, 515 ... Recording media

Japanese Unexamined Patent Publication No. 2003-150940

Claims (10)

An acquisition method for acquiring multiple images of an object taken from different viewpoints,
A calculation means for calculating the position of the center of gravity of an object in the image based on a partial region of the image.
An information processing device including a determination means for determining whether or not concealment of an object has occurred in each of the images based on a characteristic value related to a change in the position of the center of gravity with time. A setting means for setting a region of interest in the image based on the position of the center of gravity of the object is further included, and the plurality of images of the different viewpoints are given in a time series composed of a plurality of frames, respectively. The information processing apparatus according to claim 1, wherein the partial region is a region of interest set for the current frame based on the position of the center of gravity calculated in the frame at a past time. The information processing apparatus according to claim 2, further comprising a selection means for excluding the concealed image and selecting at least two images to be used in the three-dimensional measurement from the plurality of images. Further including a measuring means for performing three-dimensional measurement based on at least two images from different viewpoints selected by the selecting means, the setting means is based on the distance of the object obtained by the three-dimensional measurement. The information processing apparatus according to claim 3, further comprising changing the size of the region of interest. The characteristic value related to the time change of the center of gravity position is a change in the center of gravity velocity, and the determination by the determination means is described in any one of claims 1 to 4 based on a threshold condition for the change in the center of gravity velocity. Information processing equipment. The characteristic value related to the time change of the center of gravity position is a change in the center of gravity acceleration, and the determination by the determination means is described in any one of claims 1 to 4 based on a threshold condition for the change in the center of gravity acceleration. Information processing equipment. Multiple imaging devices, each located at a different position,
An acquisition means for acquiring a plurality of images taken by the plurality of imaging devices, and
A calculation means for calculating the position of the center of gravity of an object in the image based on a partial region of the image.
An information processing system including a determination means for determining whether or not concealment of an object has occurred in each of the images based on a characteristic value related to a change in the position of the center of gravity with time. The information processing system according to claim 7, wherein each of the image pickup devices is a high-speed vision camera that performs shooting in timing synchronization based on a signal from the synchronization device. It is an information processing method, and the computer
The step of acquiring multiple images of an object taken from different viewpoints,
A step of calculating the position of the center of gravity of an object in the image based on a partial region of the image,
An information processing method that executes a step of determining whether or not concealment of an object has occurred in each of the images based on a characteristic value related to a change in the position of the center of gravity with time. Computer,
Acquisition means for acquiring multiple images of an object taken from different viewpoints,
Concealment of the object occurs in each of the images based on the calculation means for calculating the position of the center of gravity of the object in the image based on the partial region of the image and the characteristic value related to the time change of the position of the center of gravity. A program for functioning as a judgment means for determining whether or not an image has been used.

Images